KR20220078420A - Hierarchical estimation method for hand poses using random decision forests, recording medium and device for performing the method - Google Patents

Abstract

A hierarchical method for estimating hand posture using a random forest includes the steps of converting a depth image expressed in a two-dimensional image coordinate system into a three-dimensional camera coordinate system and pre-processing; generating a random forest for the learned palm posture using a hand model, a depth image, and a GT posture as input data; generating a random forest for the learned finger posture depending on the learned palm posture; estimating a palm posture including information on global rotation using a random forest for the learned palm posture; estimating a finger posture using a random forest for the learned finger posture dependent on the learned palm posture; and estimating a final hand posture by adding the estimated palm posture and finger posture. Accordingly, it is possible to accurately and quickly estimate the hand posture.

Description

Hierarchical estimation method of hand posture using random forest, recording medium and apparatus for performing the same

The present invention relates to a hierarchical method for estimating hand posture using a random forest, a recording medium and an apparatus for performing the same, and more particularly, to first estimate a palm including information on global rotation, and based on the result, each It relates to a technique for learning a random forest and estimating hand posture with a hierarchical procedure for estimating fingers individually.

For human computer interaction (HCI), a mouse and keyboard have been generally used. However, as interest in HCI user interfaces increases, the importance of research on hand postures and gestures to provide a natural input method without using additional equipment is increasing.

The prior art describes the human hand as the most natural, effective, and important interactive tool for HCI applications. The hand is a part of the body that can perform a variety of complex tasks, and has potential to be used in applications such as keyboard-free air input, virtual reality, object manipulation in augmented reality, games, sign language recognition, robot manipulation, and remote surgery. evaluated as having

The study of hand postures and gestures in computer vision is a field in which various approaches have been continuously proposed. However, it is classified as a difficult field due to various problems with the hands.

Research related to hand posture and hand gestures is a research field in which various approaches have been proposed over the past 20 years. In the meantime, most studies have been conducted on images acquired using RGB cameras. Since RGB images do not contain 3D information, segmenting the hand region itself was an important problem, and estimating the hand posture was treated as a more difficult problem due to the complexity of kinematics, environmental dynamics, and background noise.

Existing sensors for gesture detection are largely divided into three types. Onboard-based sensors use accelerometers or gyro sensors to capture hand and finger movements. The multi-touch screen sensor is mainly used in mobile devices and has the disadvantage of limiting the distance between the user and the computer. Vision-based sensors use cameras.

Therefore, the operating distance is much longer than that of the multi-touch screen sensor, and there is no physical contact with the user, so it has the advantage of being more natural than the onboard-based sensor, but there is a problem of high computational complexity. In addition, these sensors do not detect the detailed shape of the finger and have a limit of detecting only a gesture.

The glove-based motion capture magnetic device causes unnatural hand movements, and there is a problem that needs to be corrected according to the user, and additional costs are incurred due to the purchase of equipment. In addition, estimation of hand posture using a depth camera increases the cost of installation, and there is a problem in that distribution is limited due to hardware limitations. In addition, marker-based equipment basically causes unnatural hand movements.

As described above, although the body posture estimation method, which has shown excellent performance in the past, is applied for hand posture estimation, there is a problem in that the result is less accurate than the body posture estimation result.

This is because the hand has a high degree of freedom, and the image change according to the rotation of the axis is relatively larger than that of the face or body. In particular, since there is a large change in rotation in the yaw direction, a technique is needed to solve the problem of shape change and magnetic occlusion due to movement and rotation caused by the high degree of freedom of the hand.

KR 10-1994311 B1 KR 10-2020-0107311 A KR 10-1853276 B1

Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, Andrew Blake, “Real-Time Human Pose Recognition in Parts from Single Depth Images”, Computer Vision and Pattern Recognition, 2011. Danhang Tang, Jonathan Taylor, Pushmeet Kohli, Cem Keskin, Tae-Kyun Kim, Jamie Shotton, “Opening the Black Box: Hierarchical Sampling Optimization for Estimating Human Hand Pose”, IEEE International Conference on Computer Vision, pp. 3325-3333, Dec 07-13, 2015. Markus Oberweger Paul Wohlhart Vincent Lepetit, “Hands Deep in Deep Learning for Hand Pose Estimation”, arXiv, 1502.06807v2, 2015.

Accordingly, it is an object of the present invention to provide a hierarchical method for estimating hand posture using a random forest.

Another object of the present invention is to provide a recording medium in which a computer program for performing the hierarchical method of estimating hand posture using the random forest is recorded.

Another object of the present invention is to provide an apparatus for performing a hierarchical method of estimating hand posture using the random forest.

A hierarchical method for estimating hand posture using a random forest according to an embodiment of the present invention for realizing the object of the present invention includes the steps of converting a depth image expressed in a two-dimensional image coordinate system into a three-dimensional camera coordinate system and pre-processing; generating a random forest for the learned palm posture using a hand model, a depth image, and a GT posture as input data; generating a random forest for the learned finger posture depending on the learned palm posture; estimating a palm posture including information on global rotation using a random forest for the learned palm posture; estimating a finger posture using a random forest for the learned finger posture dependent on the learned palm posture; and estimating a final hand posture by adding the estimated palm posture and finger posture.

In an embodiment of the present invention, generating a random forest for the learned palm posture; generating a random forest for the finger posture; estimating the palm posture; And the step of estimating the finger posture may be performed hierarchically by repeating each N (here, N is a natural number).

In an embodiment of the present invention, the steps of generating a random forest for the learned palm posture and generating a random forest for the finger posture include, respectively, a hand model and a GT posture for learning using a transformation matrix. Model alignment step of aligning with the depth image; A random forest learning step of extracting a spherical three-dimensional offset feature and learning and outputting a random forest in the form of selecting a feature that minimizes the variance of the residual of the aligned hand model and GT posture; a posture estimation step of extracting a spherical three-dimensional offset feature using the learned N-th random forest and estimating the residual of the hand model; and a model update step of transforming the hand model by reflecting the estimated residual to the hand model.

In an embodiment of the present invention, the updating of the model for transforming the hand model by reflecting the estimated residual to the hand model may further include updating the transformation matrix for the N+1-th learning.

In an embodiment of the present invention, the step of estimating the palm posture and estimating the finger posture may include, respectively, a model alignment step of aligning a hand model and a depth image for learning using a transformation matrix; a posture estimation step of extracting a spherical three-dimensional offset feature using the learned N-th random forest and estimating the residual of the hand model; and a model update step of transforming the hand model by reflecting the estimated residual to the hand model.

In an embodiment of the present invention, the posture estimation step may estimate a residual between the currently aligned hand posture and the hand posture to be aligned next using the learned random forest.

In an embodiment of the present invention, the posture estimating step includes: tree nodes constituting the random forest reaching leaf nodes through branching; calculating a feature value from an input depth image using a three-dimensional offset feature assigned to a node for branching; repeatedly performing a branching process to a child node through comparison with a threshold value assigned to the node; and estimating a residual value when a leaf node is reached through branching.

In an embodiment of the present invention, the posture estimating step is performed on all trees constituting the random forest; and averaging the residual values estimated by all reached leaf nodes and using them as the finally estimated residual values.

In an embodiment of the present invention, the transformation matrix aligns the hand model with the point cloud, aligns the spherical three-dimensional offset feature with the camera coordinate system, and aligns the residual estimated in the step of estimating the posture with the camera coordinate system in the model update step You can transform the hand model.

In a computer-readable storage medium according to an embodiment for realizing another object of the present invention, a computer program for performing a hierarchical method of estimating a hand posture using the random forest is recorded.

A hierarchical apparatus for estimating hand posture using a random forest according to an embodiment for realizing another object of the present invention includes: a preprocessing unit for converting a depth image expressed in a two-dimensional image coordinate system into a three-dimensional camera coordinate system; a palm posture learning unit that outputs a random forest for palm postures learned using a hand model, a depth image, and a GT posture as input data; a finger posture learning unit for outputting a random forest for the learned finger posture depending on the learned palm posture; a palm posture estimator for estimating a palm posture including information on global rotation using a random forest for the learned palm posture; a finger posture estimator for estimating a finger posture using a random forest for the learned finger posture dependent on the learned palm posture; and a hand posture expression unit for estimating a final hand posture by adding the estimated palm posture and finger posture.

In an embodiment of the present invention, the palm posture learning unit; the finger posture learning unit; the palm posture estimation unit; And, learning and estimation of the finger posture estimator may be performed hierarchically by repeating each N (here, N is a natural number).

In an embodiment of the present invention, the palm posture learning unit and the finger posture learning unit include: a model aligning unit for aligning the hand model and the GT posture with the depth image for learning using a transformation matrix, respectively; a random forest learning unit that extracts a spherical three-dimensional offset feature and selects a feature that minimizes the variance of the aligned hand model and GT posture residual by learning and outputting a random forest; a posture estimator that extracts a spherical three-dimensional offset feature using the learned N-th random forest and estimates the residual of the hand model; and a model updater that transforms the hand model by reflecting the estimated residual to the hand model, and updates the transformation matrix for the N+1th learning.

In an embodiment of the present invention, the palm posture estimator and the finger posture estimator may include a model aligner for aligning the hand model and the depth image together for learning using a transformation matrix, respectively; a posture estimator that extracts a spherical three-dimensional offset feature using the learned N-th random forest and estimates the residual of the hand model; and a model updater configured to transform the hand model by reflecting the estimated residual to the hand model.

In an embodiment of the present invention, the posture estimator, the tree nodes constituting the random forest reach the leaf node through branching, and use the 3D offset feature assigned to the node for branching to the feature value in the input depth image. The process of estimating the residual value when reaching a leaf node through branching by repeatedly performing the branching process to the child node through comparison with the threshold assigned to the node By performing it on the tree, the residual values estimated by all reached leaf nodes can be averaged and used as the final estimated residual.

According to the hierarchical estimation method of hand posture using such a random forest, the high-order degree of freedom, shape change, and occlusion problems of hand posture can be solved through a hierarchical estimation method that handles palms and fingers individually using an inverse transformation matrix. . In addition, it is possible to accurately and quickly estimate the hand posture by solving the real-time condition problem through the random forest using simple features.

1 is a block diagram of a hierarchical apparatus for estimating a hand posture using a random forest according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating examples of hierarchical hand posture estimation of FIG. 1 .
3 is a view showing image coordinate system data and camera coordinate system data.
4 is a view showing the shape of various boundary volumes.
5 is a view showing the results of visualization of three vectors obtained from the direction bounding box applied to the point cloud on the camera coordinate system.
6 is a view showing a palm model before alignment.
7 is a diagram showing the converted palm model and the aligned palm model.
8 is a diagram showing a finger model transformed using an inverse transformation matrix.
9 is a diagram showing a finger model aligned using an inverse transformation matrix.
10 is a diagram showing a two-dimensional offset feature used to find each part of the body.
11 is a diagram showing a spherical three-dimensional offset feature.
12 is a diagram for explaining a random forest.
13 is a diagram illustrating an estimated palm posture.
14 is a diagram illustrating an estimated finger posture.
15 is a flowchart of a hierarchical method of learning a hand posture according to an embodiment of the present invention.
16 is a flowchart of a method for estimating a hand posture according to an embodiment of the present invention.
17 is a graph showing the error with respect to the number of iterations of the hierarchical estimation of the present invention.
18 is a graph showing the results of quantitative evaluation of the average joint error of the present invention and the prior art.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents as those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a hierarchical apparatus for estimating a hand posture using a random forest according to an embodiment of the present invention.

The hierarchical apparatus for estimating hand posture using a random forest according to the present invention (10, hereinafter apparatus) performs hierarchical learning and estimation in order to solve the problem of shape change and self-occlusion due to movement and rotation caused by the high-dimensional freedom of the hand. present.

Referring to FIG. 1 , the device 10 according to the present invention includes a preprocessor 100 , a palm posture learning unit 200 , a finger posture learning unit 300 , a palm posture estimating unit 400 , and a finger posture estimating unit ( 500) and a hand posture expression unit 700 .

In the device 10 of the present invention, software (application) for performing hierarchical estimation of hand posture using random forest may be installed and executed, and the preprocessor 100, palm posture learning unit 200, and finger posture learning The configuration of the unit 300 , the palm posture estimator 400 , the finger posture estimator 500 , and the hand posture expression unit 700 allows hierarchical estimation of hand posture using the random forest executed in the device 10 . It can be controlled by software to perform.

The device 10 of the present invention may be a separate terminal or a part of a module of the terminal. In addition, the configuration of the preprocessor 100, the palm posture learning unit 200, the finger posture learning unit 300, the palm posture estimating unit 400, the finger posture estimating unit 500 and the hand posture expression unit 700 is It may be formed as an integrated module, or may consist of one or more modules. However, on the contrary, each configuration may be formed of a separate module.

The device 10 of the present invention may be mobile or stationary. The apparatus 10 of the present invention may be in the form of a server or an engine, and may include a device, an application, a terminal, a user equipment (UE), a mobile station (MS), It may be called by other terms such as a wireless device, a handheld device, and the like.

The device 10 of the present invention may execute or manufacture various software based on an operating system (OS), that is, the system. The operating system is a system program for software to use the hardware of the device, and is a mobile computer operating system such as Android OS, iOS, Windows Mobile OS, Bada OS, Symbian OS, Blackberry OS and Windows series, Linux series, Unix series, It can include all computer operating systems such as MAC, AIX, and HP-UX.

In the present invention, the hierarchical learning and estimation method is proposed to solve the problem of shape change and magnetic occlusion due to movement and rotation caused by the high degree of freedom of the hand. This problem arises from the global hand position and rotation, and the individual movements of the fingers.

In general, the global position and rotation of the hand is expressed as the global position and rotation of the wrist. The posture of the fingers depends on the global position and rotation of the wrist, and each of the five fingers has an individual position and movement.

In the present invention, 21 total hand joints are divided into 6 palmar joints and 15 finger joints. And the wrist joint including the global rotation information of the hand is included in the palm joint. The palm is simpler than the fingers and may be easier to guess because the shape change is small.

Therefore, the present invention proposes a method of estimating a palm including information on global rotation first, learning a random forest with a hierarchical procedure of estimating each finger individually based on the result, and estimating a hand posture . The learning and estimation procedures for palm posture and finger posture can be performed hierarchically by repeating the number of repetitions N designated by the user.

In addition, a hand model is used to represent the estimated hand posture. The hand model is aligned with the depth image using a transformation matrix and transformed through an estimation process using a random forest to express the final hand posture.

Random forest is used to solve the real-time condition problem. In addition, a spherical three-dimensional offset feature is proposed for training and estimation of random forests. A random forest is an ensemble of trees composed of multiple trees. Each tree consists of several split nodes and leaf nodes.

The information input to the tree reaches the leaf node through the branching of the split node, and the leaf node estimates the information about the hand posture. The operation performed for the branching of the split node is a simple operation that compares the magnitude of the feature value and the threshold value, and this process is repeated until a leaf node is reached.

Accordingly, there is an advantage in that the hand posture can be estimated very quickly. However, there is a problem in that the consumption time increases exponentially as the amount of data increases during learning. However, since the learning process is performed offline, it does not affect the estimation speed. The spherical three-dimensional offset feature is used as a feature for training and estimation of the random forest. In general, we propose a method for extending features extracted from 2D flat images into 3D space.

The learning units 200 and 300 hierarchically learn the random forest. The learned random forest is used for estimating the hand posture in the estimators 400 and 500 . A hand model, a depth image, and a GT posture are used as input data of the learning units 200 and 300 , and N learned random forests are output.

Except for the preprocessing step, the model alignment, random forest learning, posture estimation, and model update steps are performed separately for the palm and fingers. In addition, the hierarchical learning process for the fingers becomes dependent on the learning results for the palm.

The input data of the learning units 200 and 300 includes a ground truth (GT) posture indicating a known joint position with respect to a posture of a hand existing in a depth image for learning.

The estimation units 400 and 500 do not use the GT posture as input data. The depth image and GT posture used in the learning units 200 and 300 are created from the synthesized data generator. First, the preprocessor 100 converts a depth image into a point cloud expressed in a three-dimensional camera coordinate system. Most of the hierarchical learning procedure proposed in the present invention is performed in the camera coordinate system.

Second, model alignment aligns the hand model and GT posture with the depth image for learning. A transformation matrix is used for alignment and is generated from the direction bounding box of the hand region.

Third, forest learning learns a random forest in the form of extracting a spherical three-dimensional offset feature and selecting a feature that minimizes the variance of the residual of the aligned hand model and GT posture. Learning can be performed as many times as N iterations specified by the user, and N learned random forests are output.

Fourth, posture estimation uses the learned N-th random forest to extract spherical three-dimensional offset features and estimate the residual of the hand model. As above, it is repeated N times as many times as specified.

Fifth, the model update transforms the hand model by reflecting the residual estimated from the posture estimation in the hand model. The modified hand model represents the estimated hand posture. And the transformation matrix is updated for the N+1th learning.

In the present invention, the transformation matrix is used throughout the learning unit and the estimation unit, and performs three major roles. The first is to align the hand model with the point cloud. The second is to align the spherical 3D offset features with the camera coordinate system. The third is to transform the hand model in the model update step by aligning the residuals estimated in the posture estimation step with the camera coordinate system.

The estimation units 400 and 500 hierarchically estimate the hand posture using the learned random forest. The hand model and depth image are used as input data of the estimation unit, and the estimated hand posture is output.

In the same manner as in the learning units 200 and 300 , all steps except the pre-processing step are performed separately for the palm and fingers. The estimation result of the finger posture is dependent on the estimation result of the palm posture, and the final estimated hand posture is output by adding two types of estimated postures. All steps of the estimation units 400 and 500 are processed the same as those of the learning units 200 and 300, and the random forest learning step of the learning unit is excluded.

Figure 2 shows an example of hierarchical hand posture estimation drawn arbitrarily for visualization. The palm posture estimation and the finger posture estimation were each repeated 3 times. First, the palm model is estimated and updated, and the finger model is estimated and updated depending on the estimated palm posture.

The learning units 200 and 300 use the synthesized depth image and the synthesized GT posture together with the hand model as input data for learning the random forest. The estimation units 400 and 500 use the hand model and the depth image obtained from the depth camera as input data for estimating the hand posture using the learned random forest.

The preprocessor 100 pre-processes the depth image expressed in the two-dimensional image coordinate system by converting it into a three-dimensional camera coordinate system.

Preprocessing is a process of converting a depth image expressed in a two-dimensional image coordinate system into a three-dimensional camera coordinate system. The depth image projected on the image plane includes distortion information of the camera.

Therefore, when dealing with information having a shape, it is known that processing in a camera coordinate system that does not include distortion information is advantageous rather than processing in an image coordinate system that includes distortion information. In addition, since the depth image has data corresponding to distance as pixel values, it is advantageous to express it in a three-dimensional camera coordinate system rather than a two-dimensional image coordinate system.

Therefore, the depth image is converted from the image coordinate system to the camera coordinate system through preprocessing. The transformed pixel set is referred to as a point cloud. Most of the subsequent steps in the learning units 200 and 300 and the estimators 400 and 500 are performed based on the camera coordinate system.

If you know the camera's internal parameters, you can convert the image coordinate system data to the camera coordinate system or convert the camera coordinate system data to the image coordinate system through matrix operation that represents the relationship. However, most of the camera internal parameters are unknown. Therefore, it is generally used by estimating camera internal parameters through camera calibration.

As an example, the camera calibration method uses a predefined camera internal parameter in a public database or obtains the camera internal parameter through a function that returns the internal camera parameter implemented in a library provided by the manufacturer of the depth camera. can

를 대응하는 카메라 좌표계상의 한 점

로 변환하는 식은 다음의 수학식 1과 같다. One pixel in the image coordinate system of a two-dimensional RGB image

a point in the camera coordinate system corresponding to

The formula for converting to is the same as Equation 1 below.

[Equation 1]

,

,

와

는 초점거리를 말하며,

와

는 주점을 의미한다. 깊이영상은 거리에 대한 데이터를 화소 값으로 가지고 있으므로, 이 정보를 포함하는 행렬표현을 사용하여 정리하면 다음과 같은 수학식 2로 깊이영상의 영상 좌표계 데이터를 카메라 좌표계로 변환할 수 있다.here,

Wow

is the focal length,

Wow

means pub. Since the depth image has distance data as pixel values, if it is summarized using a matrix expression including this information, the image coordinate system data of the depth image can be converted into the camera coordinate system by Equation 2 below.

[Equation 2]

는 영상 좌표계에 표현된 깊이영상의 한 화소를 의미하고,

는 그에 대응하는 카메라 좌표계상의 한 점을 나타낸다. 그리고,

는 깊이영상의 화소 값, 즉 거리를 나타내는 깊이 값이다.in Equation 2

means one pixel of the depth image expressed in the image coordinate system,

denotes a point on the camera coordinate system corresponding to it. and,

is a pixel value of the depth image, that is, a depth value indicating distance.

3 is the result of converting the hand depth image projected on the image coordinate system to the camera coordinate system, and visualized using Matlab. The red hand in the lower right part represents data in the image coordinate system, and the black hand in the upper left part represents data converted to the camera coordinate system.

First, the model alignment step of aligning the hand model and the point cloud will be described. After the hand model is aligned with the point cloud, it is transformed through an estimation process, and the transformed hand model expresses the posture the hand is taking.

The hand posture estimation process is performed as a hierarchical procedure of estimating the palm posture first and estimating the finger posture depending on the result. To suit this hierarchical estimation procedure, the model alignment step aligns the palm model first, and then aligns the finger model depending on the estimated palm posture.

The inverse transformation matrix is used to align the hand model. In general, a left-handed coordinate system and a right-handed coordinate system are used to describe a three-dimensional coordinate system. In both coordinate systems, the direction indicated by the index finger indicates the Y axis, the direction of the thumb indicates the X axis, and the direction of the middle finger indicates the Z axis. If the coordinate system can be estimated from the point cloud of the hand, each axis can represent the approximate rotation information of the hand.

A transformation matrix and an inverse transformation matrix are generated from the estimated coordinate system, and the hand model is aligned with the point cloud through the inverse transformation matrix. The inverse transformation matrix that aligns the palm model is created using a direction bounding box. And the inverse transformation matrix that aligns the finger model is generated from the estimated palm posture.

An inverse transformation matrix for aligning the palm model is generated using a bounding volume approach. The boundary volume approach is used to roughly estimate the coordinate system by finding the principal direction of the point cloud. Boundary volume methods include a spherical volume, an axis-aligned bounding box, a directional bounding box, a convex hull, and the like, and an example of each method is shown in FIG. 4 . Each boundary volume method can be extended in three dimensions, and time complexity and spatial density generally increase from left to right.

Spherical volumes and convex hulls for point clouds are difficult to define. Since the axis alignment bounding box is aligned on the basis of the coordinate axis, there is a problem in that the main direction is expressed only as an axis. Therefore, the direction bounding box method that can express information about the main direction of the point cloud with the longest side is used. The coordinate system including the circumferential direction information of the point cloud is calculated from the vector representing each side of the box. Also, using vectors, we create a transformation matrix and an inverse transformation matrix for aligning the palm model.

The point cloud transformed into the camera coordinate system may include outliers. Outliers can be caused by the performance of the depth camera and the surrounding environment, and adversely affect the outcome of the directional bounding box as it affects the size of the bounding volume. Therefore, in order to remove the outlier, a radial outlier removal and a statistical outlier removal method may be applied. Then, for the filtered point cloud, a direction bounding box is applied to roughly estimate the principal direction of the point cloud.

를 정규화 하여 기저벡터

를 계산한다. From the direction bounding box, three vectors orthogonal to each other with the center of gravity as the starting point can be extracted. Each vector represents the axis of the direction bounding box, and since it has a property of being orthogonal to each other, it can be regarded as a coordinate system by itself. The transformation matrix is created using the properties of vectors considered as coordinate systems. The process of creating a transformation matrix is as follows. First, the vector extracted from the direction bounding box as shown in Equation 3 below

by normalizing the basis vector

to calculate

[Equation 3]

을 다음의 수학식 4와 같이 생성한다. After that, the basis vectors are arranged in the form of a matrix, and the transformation matrix for the coordinate system is

is generated as in Equation 4 below.

[Equation 4]

에 대한 역행렬

으로 계산된다. 하지만 앞서 산출된 기저벡터는 서로 직교하는 성질을 가지고 있기 때문에 서로의 내적은 항상 0이 된다. 그러므로 변환행렬

의 역행렬

은 전치행렬

로써 아래의 수학식 5와 같이 표현된다.The inverse transformation matrix for aligning the hand model is the transformation matrix

inverse matrix for

is calculated as However, since the basis vectors calculated above are orthogonal to each other, the dot product of each other is always 0. Therefore, the transformation matrix

inverse of

is the transpose matrix

It is expressed as Equation 5 below.

[Equation 5]

와

, 그리고

와

로 표기된다.A transformation matrix and an inverse transformation matrix are generated for the palm and the finger, respectively, and each

Wow

, and

Wow

is marked with

5 shows the visualization result of three vectors obtained from the direction bounding box applied to the point cloud on the camera coordinate system. The center of gravity of the direction bounding box is indicated by a red dot. The longest vector obtained by applying the direction bounding box is set as the principal Y-axis and is shown in red. The blue vector pointing to the right is orthogonal to the X and X-Y axes, and the vector pointing to the palm is the Z axis and is drawn in green. In FIG. 5, each vector is enlarged for visual expression.

의 손바닥 관절을 손바닥 모델

로 표기하며, 이것을 단순히 좌표계에 시각화 하여 표현하면 도 6과 같이 빨간색 별 모양으로 그려진다. hand model

the palmar joint of the palm model

, and if it is expressed simply by visualizing it in the coordinate system, it is drawn in the shape of a red star as shown in FIG. 6 .

에 손바닥 모델

을 곱하면 기저벡터로 구성된 좌표계로 손바닥 모델의 좌표가 아래의 수학식 6과 같이 변환된다.Inverse transformation matrix to align the hand model with the point cloud

on palm model

When multiplied by , the coordinates of the palm model are transformed as shown in Equation 6 below into a coordinate system composed of basis vectors.

[Equation 6]

은

의 좌표를 기준으로 변환되었기 때문에 도 7과 같이 하단부에 위치하는 빨간색 별 모양으로 그려진다. The palm model with the transformed coordinate system

silver

Since it is converted based on the coordinates of , it is drawn in the shape of a red star located at the lower part as shown in FIG. 7 .

의 정렬중심을 중지의 MCP 관절인

로 설정하고 방향 경계 상자의 중심점

와의 거리

를 계산한다. 그 후,

를 모든 관절에 대해 더해주면 정렬된 손바닥 모델

를 아래의 수학식 7과 같이 얻을 수 있다. So, to align with the point cloud

Alignment of the center of the middle of the MCP joint

set to and the center point of the direction bounding box

distance from

to calculate After that,

If we add for all joints, we get an aligned palm model.

can be obtained as in Equation 7 below.

[Equation 7]

와

의 각 관절 사이에 직선을 임의로 추가하여 하였으며, 모든 관절을 향해 직선이 뻗어 나오는 원점이 손목 관절에 해당한다.The finally aligned hand model is displayed as a green dot overlaid with the point cloud in the figure above. In order to easily express the relationship between the palm joints

Wow

This was done by adding a straight line arbitrarily between each joint of , and the origin of the straight line extending toward all joints corresponds to the wrist joint.

을 사용하여 생성된다. 즉, 아래의 수학식 8과 같이

은

와 동일하다.The transformation matrix and inverse transformation matrix for finger model alignment are palm transformation matrices used in the palm posture estimation process instead of the direction bounding box.

is created using That is, as in Equation 8 below

silver

same as

[Equation 8]

는 손가락 모델을 정렬하기 위한 역변환행렬

를 사용하여 아래의 식과 같이 행렬연산을 통해 아래의 수학식 9와 같이 변환된다.The finger model is aligned in a similar way to the palm model. finger model

is the inverse transformation matrix for aligning the finger model.

is transformed as in Equation 9 below through matrix operation as shown in the following equation using

[Equation 9]

만 존재하기 때문이다. The converted finger model shows an incorrect alignment result in which the five finger models are aligned together based on the finger origin as shown in FIG. 8 . The reason is that one inverse transformation matrix for all finger joints

only because it exists.

를 구성하는 MCP에 해당하는 관절들

을 정렬의 중심으로 하여

를 구성하는 각 손가락 모델을 이동시켜 손가락을 개별적으로 아래의 수학식 10과 같이 정렬하고, 그 결과는 도 9와 같다.Thus, the estimated palm model

Joints corresponding to the MCP constituting

as the center of sorting

By moving each finger model constituting the , the fingers are individually aligned as in Equation 10 below, and the result is shown in FIG. 9 .

[Equation 10]

Hereinafter, a random forest learning step performed by the learning units 200 and 300, a posture estimation step of estimating a posture using the learned random forest, and a method of extracting features used for learning and estimation will be described.

Random forests are used to solve real-time conditional problems. The random forest simply performs an operation that compares the magnitude of the feature value and the threshold value for node branching. Therefore, there is an advantage of estimating the hand posture very quickly. Feature values are extracted for training and estimation of random forests.

A two-dimensional offset feature is commonly used for learning random forests. However, since the depth image has a three-dimensional distance value, the present invention proposes a spherical three-dimensional offset feature that extends the two-dimensional feature into three dimensions.

An offset feature similar to a feature generally used in random forest learning and posture estimation (eg, non-patent document 1 of the prior art document) is mainly used. These features individually provide only a very weak level of feature values to perform accurate classification. However, when feature values are combined through a random forest, different types of hand postures can be accurately estimated.

는 특징이 참조되는 화소의 원점을 나타내고 노란색 엑스 모양으로 표기된다. 그리고 특징 매개변수

는 영상에 대한 2차원 오프셋을 나타내며

로 정의된다.

과

는 각각 빨간색 동그라미로 표기된다. 특징에 대한 응답 값은 아래의 수학식 11과 같이 계산된다. 10 is a diagram referenced in Non-Patent Document 1 of the prior art document and shows the two-dimensional offset feature used to find each part of the body. position of a given reference pixel

denotes the origin of the pixel to which the feature is referenced and is denoted by a yellow X. and feature parameters

represents the two-dimensional offset to the image,

is defined as

class

Each is indicated by a red circle. The response value for the feature is calculated as in Equation 11 below.

[Equation 11]

는 특정한 영상내의 화소

의 위치에 해당하는 깊이 값을 나타낸다. 즉, 수학식 11로 계산되는 특징 값은 깊이영상에서 두 오프셋 위치에 해당하는 화소가 가지는 깊이 값의 차이이다. 깊이카메라로 촬영된 객체는 카메라로부터의 거리에 따라 깊이 영상에서 서로 다른 화소 값을 가진다. function

is a pixel in a particular image.

Indicates the depth value corresponding to the position of . That is, the feature value calculated by Equation 11 is the difference between the depth values of the pixels corresponding to the two offset positions in the depth image. Objects photographed with the depth camera have different pixel values in the depth image according to the distance from the camera.

에 의한 정규화는 특징응답이 깊이 불변임을 보장한다. 만약 오프셋이 배경 또는 영상좌표 외부에 존재하는 경우 선행기술문헌의 비특허문헌 1과 동일하게 큰 양의 상수 값을 지정한다. 트리를 학습하는 동안 오프셋은 고정된 크기의 영역 내에서 랜덤하게 선택 된다. in Equation 11

Normalization by , guarantees that the feature response is depth invariant. If the offset exists outside the background or image coordinates, a large positive constant value is designated as in Non-Patent Document 1 of the prior art document. During tree learning, the offset is randomly selected within a fixed-sized region.

경우에 대해 위의 식을 적용하면 (a)에서는 큰 값을 가지게 되지만 (b)에서는 작은 값을 가지게 되고 신체부위와 배경을 구분하기 위한 특징으로 사용할 수 있다. 유사한 이유로

매개변수를 사용하는 경우 팔과 같은 얇은 구조를 찾아내는데 도움이 될 수 있다. 10 shows two types of features. parameter

If the above formula is applied to the case, it has a large value in (a), but a small value in (b), and can be used as a feature to distinguish body parts and backgrounds. for similar reasons

When using parameters, it can help to spot thin structures such as arms.

This simple depth difference feature shows very good computational efficiency. No pre-processing is required, each feature only uses up to 3 image pixels and only performs arithmetic operations up to 5 times to obtain feature values. And this process has the advantage that it can be processed in parallel within the random forest.

The offset feature described above is performed on a 2D image. Although a depth image is expressed in an image in a two-dimensional space, it can be regarded as three-dimensional data because it has distance information as a pixel value. Therefore, we propose a method of extending the space for offset feature extraction from 2D to 3D.

반지름이

인 구의 방정식은

이다. First, we limit the feature extraction region using a spherical equation to define the space for extracting the position of the offset in the three-dimensional space. center

radius

The equation for the population is

to be.

이고, 각 축의 범위를 -1부터 1 사이로 정규화한 3차원 오프셋 추출 공간을 기본설정으로 사용한다. 따라서, 구의 방정식은

이 된다. 그 후

각각에 대해 -1부터 1사이의 값을 랜덤함수를 사용하여 랜덤하게 할당한다. In the present invention, the center

and a three-dimensional offset extraction space in which the range of each axis is normalized from -1 to 1 is used as a default setting. So, the sphere's equation is

becomes this After that

For each, a value between -1 and 1 is randomly assigned using a random function.

의 값이

을 만족하면 생성된 랜덤 값이 구 공간 내에 포함되므로 3차원 오프셋 특징의 특징 매개변수

로 사용될 수 있다. 즉

일 때,

이 된다. 특징 추출을 위한 공간의 크기를 변경 하려면

값의 크기를 변경한다. The three-dimensional offset feature is set in the space of the sphere as shown in FIG. 11 by the equation of the sphere. If assigned

the value of

If is satisfied, the generated random value is contained within the sphere space, so the feature parameter of the three-dimensional offset feature

can be used as In other words

when,

becomes this To change the size of the space for feature extraction

Change the size of the value.

The RGB image captured by the camera has a problem in that some information about the three-dimensional space is lost because all points on the projection ray are expressed as one point. This problem still exists in the depth image taken using the depth camera.

위치에 데이터가 존재하지 않을 수 있기 때문에 완벽한 오프셋 특징 값을 추출하지 못하는 문제가 발생한다. Therefore, although the point cloud of the hand projected on the camera coordinate system contains a part of the 3D information, it does not represent the complete 3D hand shape. Therefore, even if a three-dimensional offset of the form above is used,

Since data may not exist at the location, a problem arises in that it is not possible to extract a perfect offset feature value.

로 정의된다. 역변환 과정의 수식은 다음의 수학식 12와 같다.Therefore, the feature value is calculated from the offset cloud point by inversely projecting the position of the 3D offset into the image coordinate system. For inverse projection, the 3D offset selected first is inversely transformed into the camera coordinate system. The inversely transformed 3D offset is

is defined as Equation of the inverse transformation process is as Equation 12 below.

[Equation 12]

는 구 공간에서 선택된 3차원 오프셋이며,

는 손 모델의 각 관절 위치를 나타내는 3차원 좌표이다.

은 역변환행렬을 의미한다. 카메라 좌표계로 변환된 3차원 오프셋은 역 투영 과정을 통해 다음의 수학식 13과 같이 영상 좌표계로 변환된다. here,

is the selected three-dimensional offset in spherical space,

is a three-dimensional coordinate indicating the position of each joint in the hand model.

is the inverse transformation matrix. The 3D offset converted to the camera coordinate system is converted into the image coordinate system as shown in Equation 13 below through the reverse projection process.

[Equation 13]

로 표현된다. 영상 좌표계상의 오프셋 위치는

로 표현된다.

은 초점거리를 의미하고

와

는 주점을 의미한다. 계산된 영상 좌표계의 오프셋 위치가 배경 밖에 존재하는 경우 해당 오프셋 위치의 최 외곽 화소의 위치로 대체한다. 그 후에 두 오프셋 화소의 깊이 차이를 아래의 수학식 14와 같이 계산하여 특징 값으로 사용한다.In Equation 13, the offset in the camera coordinate system is

is expressed as The offset position in the image coordinate system is

is expressed as

is the focal length

Wow

means pub. If the calculated offset position of the image coordinate system exists outside the background, it is replaced with the position of the outermost pixel of the offset position. Thereafter, the difference in depth between the two offset pixels is calculated as in Equation 14 below and used as a feature value.

[Equation 14]

는 영상 좌표계의 오프셋 위치

의 깊이 값에 해당하며

는 두 오프셋의 깊이 값의 차이이다.

is the offset position of the image coordinate system

corresponds to the depth value of

is the difference between the depth values of the two offsets.

A tree is one of the classification methods that have been widely used for a long time. However, it is known to have problems with overfitting. Trees can estimate well on pre-trained data, but perform poorly on unknown data.

의 앙상블 형태로 표현된다.

을 사용하여 트리를 구성하는 다수의 노드 중에서 특정한 노드를 정의한다. 그리고,

을 사용하여 구체적인 리프노드를 정의한다. A random forest is a tree composed of a split node (blue) and a leaf node (green) at the end as shown in FIG.

expressed in the form of an ensemble of

is used to define a specific node among many nodes composing the tree. and,

to define a specific leaf node.

로 정의되는 약한 학습자를 포함한다. 첫 번째 매개변수는

로 정의된다. 여기서

는 특징추출에 사용되는 오프셋 특징을 나타낸다. 두 번째 매개변수

는 스칼라 임계값이다. 깊이영상에서 참조되는 화소

에 대한 추정 값을 얻기 위해 약한 학습자 함수를 반복적으로 평가한다. 이 과정은 루트에서 시작하여 리프노드에 도달 할 때까지 경로를 순회하며 반복된다.Each split node has a parameter

Includes weak learners defined as the first parameter is

is defined as here

denotes an offset feature used for feature extraction. second parameter

is a scalar threshold. Pixels referenced in the depth image

Iteratively evaluates the weak learner function to obtain an estimate for This process repeats, starting at the root and traversing the path until reaching a leaf node.

의 출력 값과 임계값

의 크기가 0과 1을 구분하는 기준이 된다. 만약

이 0으로 평가 되면 노드

의 왼쪽 자식노드 방향으로 분기하고 1로 평가되면 오른쪽 자식노드 방향으로 분기한다.Whether to branch from a specific node to a child node in which direction is determined by Equation 15 below. Here, the results of the right term represent 0 and 1. function

output value and threshold of

The size of is the criterion for distinguishing 0 and 1. what if

If this evaluates to 0, the node

Branch in the direction of the left child of , and if evaluated as 1, branch in the direction of the right child.

[Equation 15]

에 도달 할 때까지 반복 수행한다. 그리고,

를 사용하여 참조되는 화소

가 도달한 특정한 리프노드를 정의한다. 동일한 절차를 각각의 트리

에 대해 깊이영상의 각 화소에 적용하여, 리프 노드의 집합이 생성되며, 아래의 수학식 16과 같이 표현된다.Do this on any leaf node

Repeat until reaching . and,

Pixels referenced using

Defines the specific leaf node reached by The same procedure for each tree

is applied to each pixel of the depth image to generate a set of leaf nodes, which is expressed as Equation 16 below.

[Equation 16]

에는 학습된 예측모델이 저장된다. 예측모델은 일반적으로 분류트리와 회귀트리를 기반으로 하는 방법을 사용한다. 분류트리의 경우 일반적으로 입력되는 데이터 집합을 특정한 부류로 할당하는데 사용된다. 예측모델은 어떠한 부류

에 대한 확률 매스 함수

이다. leaf node of each tree

The learned predictive model is stored in Predictive models generally use methods based on classification trees and regression trees. In the case of a classification tree, it is generally used to assign an input data set to a specific class. What kind of predictive model is

probability mass function for

to be.

For regression trees, we can use the relative number of votes for each weighted joint. In the present invention, the average of the residual between the position of the ground truth joint of the corresponding joint and the aligned hand joint for all training images is used as an estimated value.

에 대해 리프노드

에 도달할 때까지 하강되며 분포

가 검색된다. 분포는 포레스트의 모든 트리에 대해 평균화 되며 아래의 수학식 17과 같이 평균하여 최종 분류 결과를 얻는다. Given a depth image, the tree of the random forest is an input pixel

about leaf nodes

It descends until it reaches

is searched for The distribution is averaged over all trees in the forest, and the final classification result is obtained by averaging as in Equation 17 below.

[Equation 17]

The learning process of the random forest included in the flowchart of the learning units 200 and 300 is as follows. First, the hand model, the entire GT posture, and the full depth image are aligned using an inverse transformation matrix, respectively. Then, the residuals are calculated for the overall GT posture and the overall aligned hand model. Then, the variance of the residuals for the entire training image is calculated. Finally, the learning process of the node is to select the feature that brings the maximum variance reduction of the residual by using the three-dimensional offset feature.

When this process is performed until a certain threshold for tree generation is reached, one learned tree is completed, and when this process is completed for all trees, a learned random forest is created.

For example, the residual for 1,000 training images is expressed as 1,000 matrices consisting of 21 rows corresponding to the number and position of joints and 3 columns indicating the coordinate values x, y, and z. By randomly generating a spherical 3D offset feature, a depth difference value is calculated for the entire image for each feature. If this process is performed for 1,000 images, the depth difference value is expressed as a 512 x 1,000 matrix.

By collecting the residuals for each joint from the entire matrix calculated earlier, we make 21 1,000 x 3 matrices. The variance is calculated for each of the 21 matrices, and one feature with the maximum variance reduction is selected using the previously calculated depth difference value as a threshold. This characteristic corresponds to the position and threshold value of the offset, which are parameters of the node.

Then, based on the threshold, the learning image is divided into a left child node and a right child node, and then the previous node learning process is repeated. Finally, when the number of images is 10 or less, the corresponding node is set as a leaf node, and the average value of the corresponding joint residual is used as the estimated residual value. This process is a process of learning one tree constituting a random forest in the present invention.

The estimation units 400 and 500 estimate the residual between the currently aligned hand posture and the hand posture to be aligned next using the learned random forest. Tree nodes composing a random forest reach leaf nodes through branches. For branching, the feature value is calculated from the input depth image using the 3D offset feature assigned to the node.

After that, the branching process to the child node is repeatedly performed through comparison with the threshold value assigned to the node. When a leaf node is reached via branching, the residual value is estimated. This process is performed for all trees constituting the random forest, and the average of the residual values estimated by all reached leaf nodes is used as the final estimated residual.

The model update consists of a process of transforming a hand model using the residuals obtained from the estimation units 400 and 500 and a process of updating a transformation matrix and an inverse transformation matrix used in the repeated hierarchical estimation process. The model update is done in different ways for the palm and fingers. The estimation of hand posture proceeds in a hierarchical order of estimating the palm first and estimating the fingers depending on the result. Therefore, the palm model update step will be described first, and the finger model update step will be described.

는 정렬된 손바닥 모델

의 각 관절위치에 더해져 손바닥 모델의 관절 위치를 변형시키고 손바닥 자세로써 표현된다. 먼저, 잔차

는 정렬된 손바닥 모델의 포인트 클라우드와 동일한 좌표계에서 더해져야 하므로 역변환행렬에

을 곱해서 카메라 좌표계로 변환해야 한다. The residual of the palm model estimated using random forest in the estimation step

is an aligned palm model

The joint position of the palm model is modified by adding it to each joint position of , and it is expressed as a palm posture. First, the residual

must be added in the same coordinate system as the point cloud of the aligned palm model, so it is added to the inverse transformation matrix.

Multiply by to convert to the camera coordinate system.

만큼 수행되기 때문에, 현재의 손바닥 모델 자세는 이전단계의 모델 자세에 의존적이게 된다. 따라서, 현재의 손바닥 자세는 이전단계의 손바닥 자세에 잔차를 더하여 아래의 수학식 18과 같이 갱신된다. The hierarchical estimation process is the number of iterations specified by the user.

, the current palm model posture becomes dependent on the model posture of the previous stage. Therefore, the current palm posture is updated as in Equation 18 below by adding the residual to the palm posture of the previous stage.

[Equation 18]

은 도 13과 같이 그려지고, 계층적 추정이 반복 될 때마다 손바닥 모델은 변형된다. The palm posture representing the joint position of the current palm model estimated by adding the residuals estimated with the random forest.

is drawn as in Fig. 13, and each time the hierarchical estimation is repeated, the palm model is transformed.

를 추정하기 위해 이전에 추정된 손바닥 자세

를 이용하여 갱신된다. Y축 벡터는 다음의 수학식 19와 같이 중지 MCP와 손목의 차 연산으로 계산한다.The palm transformation matrix is the current palm posture.

Previously estimated palm posture to estimate

is updated using The Y-axis vector is calculated by calculating the difference between the middle MCP and the wrist as shown in Equation 19 below.

[Equation 19]

The Z-axis vector is calculated as the cross product of the average vector of the MCP joints and the Y-axis. The average vector of the MCP is obtained as in Equation 20 below by summing and averaging the five vectors obtained by the difference calculation between each MCP joint and the wrist joint.

[Equation 20]

Finally, the X-axis vector is calculated as in Equation 21 below by cross-producting the Y-axis vector and the Z-axis vector.

[Equation 21]

The obtained three axis vectors are calculated by normalizing the three vectors to calculate the basis vector. After that, if they are arranged in the form of a matrix, a transformation matrix is created. The inverse transformation matrix is expressed as a transpose matrix because the three basis vectors are orthogonal to each other.

은 정렬된 손가락 모델

와 동일한 좌표계에서 더해지기 위해 역변환행렬을 사용하여 변환된다. 그 후 이전 단계의 손가락 자세에 더해져서 변형된 손가락 자세를 다음의 수학식 22와 같이 나타낸다.The finger model transformation process is performed in the same way as the palm model transformation process. Residuals of the estimated finger model

silver aligned finger model

It is transformed using an inverse transformation matrix to be added in the same coordinate system as . Thereafter, the modified finger posture by adding to the finger posture of the previous step is expressed as in Equation 22 below.

[Equation 22]

는 도 14와 같이 그려지고, 계층적 추정이 반복 수행될 때마다 손가락 자세는 변형된다.

는 다섯 개의 손가락으로 구성되며, 각 손가락은 세 개의 관절 PIP, DIP, TIP로 구성된다.The finger transformation matrix is updated using the previously estimated finger posture for the next hierarchical estimation. estimated finger posture

is drawn as shown in FIG. 14, and each time the hierarchical estimation is repeatedly performed, the finger posture is changed.

is composed of five fingers, and each finger is composed of three joints PIP, DIP, and TIP.

The Y-axis vector is calculated as in Equation 23 below by calculating the difference between each MCP and each PIP as follows.

[Equation 23]

의 평균벡터와 Y축의 외적으로 계산된다.

의 평균벡터는 손가락을 구성하는 세 관절과 MCP 관절의 차 연산으로 계산된 세 개의 벡터를 더한 뒤 평균하여 연산으로 다음의 수학식 24와 같이 계산된다.The x-axis vector is

It is calculated as the cross product of the mean vector of and the Y axis.

The average vector of is calculated as in Equation 24 below by adding and averaging three vectors calculated by the difference operation between the three joints constituting the finger and the MCP joint.

[Equation 24]

The Z-axis vector is calculated as in Equation 25 below by cross-producting the Y-axis vector and the X-axis vector.

[Equation 25]

After that, a transformation matrix is generated through a normalization process, and the inverse transformation matrix is created as a transpose matrix.

The hand posture expression unit 700 estimates and outputs the final hand posture by adding the estimated palm posture and finger posture.

Accordingly, the present invention can solve the high-order degree of freedom of hand posture, shape change, and occlusion problems through a hierarchical estimation method that separately handles palms and fingers using an inverse transformation matrix. In addition, it is possible to accurately and quickly estimate the hand posture by solving the real-time condition problem through the random forest using simple features.

15 is a flowchart of a hierarchical method of learning a hand posture according to an embodiment of the present invention. 16 is a flowchart of a method for estimating a hand posture according to an embodiment of the present invention.

The hierarchical method for estimating the hand posture using the random forest according to the present embodiment may proceed in substantially the same configuration as the apparatus 10 of FIG. 1 . Accordingly, the same components as those of the device 10 of FIG. 1 are given the same reference numerals, and repeated descriptions are omitted.

In addition, the hierarchical method for estimating the hand posture using the random forest according to the present embodiment may be executed by software (application) for performing the hierarchical estimation of the hand posture using the random forest.

Referring to FIG. 15 , in the hierarchical estimation method of hand posture using a random forest according to the present embodiment, a depth image expressed in a two-dimensional image coordinate system is converted into a three-dimensional camera coordinate system and pre-processed (step S10). The input data may be a depth image and a hand model.

A random forest for the learned palm posture is generated using the hand model, depth image, and GT posture as input data, and a random forest is generated for the learned finger posture depending on the learned palm posture.

Specifically, first, a random forest is generated for the palm and learned. Align the palm model and GT posture with the depth image for learning using the transformation matrix (step S21).

A spherical three-dimensional offset feature is extracted, and a random forest of the palm is learned and output in the form of selecting a feature that minimizes the variance of the residual of the aligned palm model and GT posture (step S22). Using the output random forest, a spherical three-dimensional offset feature is extracted and the residual of the palm model is estimated (step S23). The model that transforms the palm model is updated by reflecting the estimated residual to the palm model, and the transformation matrix is updated for the N+1th learning (step S24).

The palm posture learning procedure may be performed hierarchically by repeating the number of repetitions N designated by the user (step S25). Accordingly, N random forests for the palm are generated.

When the learning of the random forest for the palm is completed, a random forest for the finger is generated and learned depending on the result.

First, the finger model and the GT posture are aligned with the depth image for learning using the transformation matrix (step S31).

The spherical three-dimensional offset feature is extracted and the random forest of the finger is learned and output in the form of selecting the feature that minimizes the variance of the residual of the aligned finger model and the GT posture (step S32). Using the output random forest, a spherical three-dimensional offset feature is extracted and the residual of the finger model is estimated (step S33). The model that transforms the finger model is updated by reflecting the estimated residual to the finger model, and the transformation matrix is updated for the N+1th learning (step S34).

The finger posture learning procedure may be performed hierarchically by repeating the number of repetitions N designated by the user (step S35). Accordingly, N random forests for the fingers are generated.

When a random forest is generated by learning the palm and fingers, the palm posture including information on global rotation is estimated based on the result, and the finger using the random forest for the learned finger posture depends on the learned palm posture. Estimate your posture

In the estimation of palm posture and finger posture, model alignment, random forest learning, posture estimation, and model update steps except for the pre-processing step are performed separately for the palm and fingers. In addition, the hierarchical learning process for the fingers becomes dependent on the learning results for the palm. The estimation step is processed in the same way as the learning process of FIG. 15, except that the random forest learning step is excluded.

Referring to FIG. 16 , after preprocessing the input data including the depth image and the hand model (step S11), the palm model and the GT posture are aligned with the depth image for learning using the transformation matrix (step S41).

Extract the spherical three-dimensional offset feature using the learned N-th random forest and estimate the residual of the palm model (step S43). The palm model is transformed by reflecting the estimated residual to the palm model (step S44).

The posture estimation step (step S43) estimates the residual between the currently aligned hand posture and the hand posture to be aligned next using the learned random forest.

Specifically, the tree nodes constituting the random forest reach leaf nodes through branching, calculating feature values from the input depth image using the 3D offset feature assigned to the nodes for branching, and assigning them to the nodes. It may include repeatedly performing a branching process to a child node through comparison with a threshold value and estimating a residual value when a leaf node is reached through branching.

The posture estimation step may be performed on all trees constituting the random forest, and the residual values estimated by all reached leaf nodes may be averaged and used as the final estimated residual.

The finger posture estimation procedure may be performed hierarchically by repeating the number of repetitions N designated by the user (step S45).

When estimation of the palm posture is completed, it is estimated for the learned finger posture depending on the estimated palm posture. The steps of estimating the finger posture (steps S51 to S55) are the same as the process of estimating the palm posture, but are performed separately.

The final hand posture is estimated by adding the estimated palm posture and finger posture.

17 is a graph showing the error with respect to the number of iterations of the hierarchical estimation of the present invention.

Referring to FIG. 17 , the error reduction results for the iteration of hierarchical estimation are as follows. When the hand model was simply aligned, the average errors were 24.22 for the palm, 23.99 for the index finger, 19.37 for the middle finger, 32.45 for the ring finger, 36.34 for the small finger, and 34.54 for the thumb.

As shown in step 1 of the graph of FIG. 17 , the estimation error was reduced to the largest width of 10 mm or more during the first estimation, and the amount of change gradually decreased as each step progressed. After the third regression, there was no change in the error or there was no significant difference, but rather the error increased. Therefore, the number of iterations for the hierarchical estimation can be set to 3 times for the palm posture and 3 times for the finger posture.

18 is a graph showing the results of quantitative evaluation of the average joint error of the present invention and the prior art.

Referring to FIG. 18 , the horizontal axis of the graph represents each finger joint. The prior art indicated by [37] is Non-Patent Document 3 of the prior art document. 1, 2, 3, 4, 5 are used in the order of thumb, index finger, middle finger, ring finger, and little finger. MCP joint is marked as M, PIP joint is marked as P, DIP joint is marked as D, and wrist joint is marked as Palm. . And, AVERAGE means the overall average. The vertical axis of the graph indicates the error and the unit is mm.

Experimental results on the mean joint error show that the hierarchical hand posture estimation method proposed in the present invention is superior to the method proposed in [37] overall. [37] contains an average error of 11.4 mm for all joints, and the proposed method contains an error of 9.72 mm.

Therefore, the experimental results of qualitative and quantitative evaluation of the present invention with the prior art show that the method proposed in the present invention accurately and quickly estimates the hand posture with an average error of 10 mm and a speed of 28 FPS.

Such a hierarchical method for estimating hand posture using a random forest may be implemented as an application or implemented in the form of program instructions that may be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer readable recording medium are specially designed and configured for the present invention, and may be known and used by those skilled in the art of computer software.

Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for carrying out the processing according to the present invention, and vice versa.

Although the above has been described with reference to the embodiments, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below You will understand.

Since the present invention showed excellent performance without being affected by changes in Kinect v2 and Soft Kinect equipment, the possibility of use in various equipment was also confirmed. In addition, a study on a method of recognizing a specific hand posture using the estimated hand posture information may also be performed.

Accordingly, the interface using the hand in human-computer interaction can be utilized in various fields such as sign language recognition, games, object manipulation in virtual reality, and remote surgery.

10: Hierarchical estimation of hand posture using random forest
100: preprocessor
200: palm posture learning unit
300: finger posture learning unit
400: palm posture estimation unit
500: finger posture estimation unit
700: hand posture expression unit

Claims (15)

pre-processing by converting a depth image expressed in a two-dimensional image coordinate system into a three-dimensional camera coordinate system;
generating a random forest for the learned palm posture using a hand model, a depth image, and a GT posture as input data;
generating a random forest for the learned finger posture depending on the learned palm posture;
estimating a palm posture including information on global rotation using a random forest for the learned palm posture;
estimating a finger posture using a random forest for the learned finger posture dependent on the learned palm posture; and
A hierarchical method of estimating a hand posture using a random forest, including; estimating a final hand posture by summing the estimated palm posture and finger posture.
According to claim 1,
generating a random forest for the learned palm posture; generating a random forest for the finger posture; estimating the palm posture; and the step of estimating the finger posture is performed hierarchically by repeating each N (here, N is a natural number).
The method of claim 2, wherein generating a random forest for the learned palm posture and generating a random forest for the finger posture comprises:
A model alignment step of aligning the hand model and the GT posture with the depth image for learning using the transformation matrix;
A random forest learning step of extracting a spherical three-dimensional offset feature and learning and outputting a random forest in the form of selecting a feature that minimizes the variance of the residual of the aligned hand model and GT posture;
a posture estimation step of extracting a spherical three-dimensional offset feature using the learned N-th random forest and estimating the residual of the hand model; and
A hierarchical method for estimating hand postures using random forests, including; a model update step of transforming the hand model by reflecting the estimated residuals in the hand model.
The method of claim 3, wherein the updating of the model by reflecting the estimated residual to the hand model to transform the hand model,
Updating the transformation matrix for the N+1th learning; hierarchical estimation method of hand posture using a random forest, further comprising a.
The method of claim 2, wherein estimating the palm posture and estimating the finger posture comprises:
A model alignment step of aligning the hand model and the depth image together for learning using a transformation matrix;
a posture estimation step of extracting a spherical three-dimensional offset feature using the learned N-th random forest and estimating the residual of the hand model; and
A hierarchical method for estimating hand postures using random forests, including; a model update step of transforming the hand model by reflecting the estimated residuals in the hand model.
According to claim 5, wherein the posture estimation step,
A hierarchical estimating method of hand posture using a random forest that estimates the residual between the currently aligned hand posture and the hand posture that should be aligned next using the learned random forest.
According to claim 5, wherein the posture estimation step,
tree nodes constituting the random forest reach leaf nodes through branching;
calculating a feature value from an input depth image using a three-dimensional offset feature assigned to a node for branching;
repeatedly performing a branching process to a child node through comparison with a threshold value assigned to the node; and
A hierarchical method of estimating hand posture using a random forest, including; estimating a residual value when a leaf node is reached through branching.
According to claim 7, wherein the posture estimation step,
performing on all trees constituting the random forest; and
A hierarchical method for estimating hand posture using a random forest, further comprising; averaging the residual values estimated by all reached leaf nodes and using it as a final estimated residual.
According to claim 1,
The transformation matrix aligns the hand model with the point cloud, aligns the spherical three-dimensional offset feature with the camera coordinate system, and aligns the residual estimated in the posture estimation step with the camera coordinate system to transform the hand model in the model update step. Hierarchical estimation of hand posture using forest.
A computer-readable storage medium having recorded thereon a computer program for performing the hierarchical method for estimating hand posture using the random forest according to claim 1 .
a preprocessor for converting a depth image expressed in a two-dimensional image coordinate system into a three-dimensional camera coordinate system;
a palm posture learning unit that outputs a random forest for palm postures learned using a hand model, a depth image, and a GT posture as input data;
a finger posture learning unit for outputting a random forest for the learned finger posture depending on the learned palm posture;
a palm posture estimator for estimating a palm posture including information on global rotation using a random forest for the learned palm posture;
a finger posture estimator for estimating a finger posture using a random forest for the learned finger posture dependent on the learned palm posture; and
A hierarchical estimating device for hand posture using a random forest, including; a hand posture expression unit for estimating a final hand posture by adding the estimated palm posture and finger posture.
12. The method of claim 11,
the palm posture learning unit; the finger posture learning unit; the palm posture estimation unit; and the learning and estimation of the finger posture estimator is hierarchically performed by repeating each N (here, N is a natural number).
The method of claim 12, wherein the palm posture learning unit and the finger posture learning unit, respectively,
a model alignment unit that aligns the hand model and the GT posture with the depth image for learning using the transformation matrix;
a random forest learning unit that extracts a spherical three-dimensional offset feature and selects a feature that minimizes the variance of the aligned hand model and GT posture residual by learning and outputting a random forest;
a posture estimator that extracts a spherical three-dimensional offset feature using the learned N-th random forest and estimates the residual of the hand model; and
A hierarchical apparatus for estimating hand postures using random forests, including; a model updater that transforms the hand model by reflecting the estimated residual to the hand model, and updates the transformation matrix for the N+1th learning.
The method of claim 12, wherein the palm posture estimating unit and the finger posture estimating unit,
a model alignment unit that aligns the hand model and the depth image together for learning using a transformation matrix;
a posture estimator that extracts a spherical three-dimensional offset feature using the learned N-th random forest and estimates the residual of the hand model; and
A hierarchical apparatus for estimating hand posture using a random forest, including; a model update unit that transforms the hand model by reflecting the estimated residual to the hand model.
15. The method of claim 14, wherein the posture estimator,
The tree nodes constituting the random forest reach the leaf node through branching, calculate the feature value from the input depth image using the 3D offset feature assigned to the node for branching, and match the threshold value assigned to the node. By repeatedly performing the branching process to child nodes through comparison, the process of estimating the residual value when reaching a leaf node through branching is performed for all trees constituting the random forest, so that all reached leaf nodes are A hierarchical device for estimating hand postures using random forests, which averages the estimated residual values and uses them as the final estimated residuals.

Images