KR20180094253A - Apparatus and Method for Estimating Pose of User - Google Patents

Abstract

Disclosed are an apparatus and a method for estimating a posture of a user. The disclosed apparatus comprises: a Kinect sensor unit arranging a plurality of Kinect sensors to surround the user and making the plurality of Kinect sensors independently take images of a skeleton structure of the user; a common coordinate system conversion unit converting the image of the skeleton structure of the user, taken by the plurality of Kinect sensors, according to a common coordinate system; an effective Kinect sensor selection unit selecting the Kinect sensors, taking images of a front side of the user, among the plurality of Kinect sensors; a weight setup unit setting up a weighting factor for each joint in the skeleton structure image of each effective Kinect sensor with respect to the selected Kinect sensors; and a skeleton integration unit integrating the skeleton structure images of the selected effective sensors based on the setup weighting factor. The weighting factor includes: a first weighting factor which is based on a track state information of the effective Kinect sensors; and a second weighting factor which is based on the direction towards the effective Kinect sensors from the user. The disclosed apparatus can stably obtain the skeleton structure image even though a view direction of the user changes by a rotation or the like.

Description

[0001] Apparatus and Method for Estimating Pose of User [0002]

Embodiments of the present invention relate to an apparatus and method for estimating a user attitude, and more particularly, to an apparatus and method for estimating a user's attitude using a plurality of Kinect sensors.

In recent years, research on virtual reality has been actively conducted, and virtual reality has been utilized in various fields such as games, entertainment, and military use.

This method of recognizing and predicting the user 's motion and attitude in virtual reality has been actively researched for interfacing in game and military training. A Kinect sensor is a typical sensor for estimating the user's posture.

The Kinect sensor is a sensor that can acquire a skeleton structure image, which is connection information for the user's joints, and is mainly used in game devices.

However, the Kinect sensor acquires the skeleton structure image of the user on the premise that the user is looking at the front of the user, and can not accurately detect the dynamic motion such as the rotation of the user.

In the virtual reality interfacing, it is required to estimate all the user's various motions and attitudes. Especially, in the case of Kunsan training, accurate posture estimation is required even if the user rotates. However, It is difficult to accurately estimate the posture.

The present invention proposes a method and apparatus for stably acquiring a user's skeleton structure image even if the direction of the user is changed by rotation or the like.

According to an aspect of the present invention, there is provided a display device comprising: a key knot sensor unit arranged to surround a plurality of kinect sensors and the plurality of kinect sensors independently acquiring the user's skeleton structure image; A common coordinate system conversion unit for converting the user's skeleton structure image acquired by the plurality of Kinect sensors into a common coordinate system; An effective keynote sensor selection unit for selecting keynote sensors for photographing the front of the user among the plurality of keynote sensors; A weight setting unit for setting weights for the joints of the skeleton structure images of the respective effective knee pick sensors for the selected effective knee pick sensors; And a skeleton synthesis unit for synthesizing a skeleton structure image of the selected effective kinect sensors based on the set weights, wherein the weight includes a first weight based on track state information of the effective kinect sensors, A user posture estimation apparatus is provided that includes a second weight based on a direction to the Kinect sensors.

The effective kinect sensor selection unit uses a forward vector orthogonal to the line connecting the left shoulder and the right finger of the user in the skeleton structure image captured by the kinect sensors.

The valid keynote sensor selection unit determines whether to photograph the front of the user through the dot product of the front vector of the previous frame and the forward vector of each keynote sensor in the current frame.

The first weight is set on a joint-by-joint basis based on state information about a joint of the skeleton structure image of each effective Kinect sensor.

Wherein the weight setting unit defines a left area and a right area based on the forward vector of the user in the skeleton structure image of the respective effective Kinect sensors and determines a direction from the user to the specific Kinect sensor Applying a predetermined second weight less than 1 to pre-established left region joints of the joints of the skeleton structure image captured by the corresponding Kinect sensor when not in the left region, A predetermined second weight less than 1 for pre-established left region joints of the joints of the skeleton structure image captured by the corresponding Kinect sensor when the direction of the sensor to the specific Kinect sensor is not in the right area Is applied.

The skeleton synthesis unit applies a weight corresponding to a product of the first weight and the second weight to each joint, and synthesizes the skeleton structure image by weighted averaging.

And an attitude estimation unit for estimating the attitude of the user based on the skeleton structure image synthesized by the skeleton synthesis unit.

According to another aspect of the present invention, there is provided a display device comprising: a key knot sensor unit arranged to surround a plurality of kinect sensors and the plurality of kinect sensors independently acquiring the user's skeleton structure image; A common coordinate system conversion unit for converting the user's skeleton structure image acquired by the plurality of Kinect sensors into a common coordinate system; An effective keynote sensor selection unit for selecting keynote sensors for photographing the front of the user among the plurality of keynote sensors; A weight setting unit for setting weights for the joints of the skeleton structure images of the respective effective knee pick sensors for the selected effective knee pick sensors; And a skeleton synthesis unit for synthesizing the skeleton structure image of the selected effective kinect sensors based on the set weights, wherein the effective kinect sensor selection unit selects, from the skeleton structure images captured by the kinect sensors, A forward vector that is orthogonal to the line connecting the shoulder and the right fishermen, and the forward vector of the previous frame and the user's posture for determining whether to photograph the forward of the user through the dot product of the forward vector for each kinetic sensor in the current frame An estimating device is provided.

According to another aspect of the present invention there is provided a method of estimating a user's attitude in an environment in which a plurality of Kinect sensors are arranged to surround a user, the plurality of Kinect sensors independently acquiring the user's skeleton structure image Step (a); (B) transforming the user's skeleton structure image acquired from the plurality of Kinect sensors into a common coordinate system; Selecting effective keynote sensors for capturing a front of the user among the plurality of keynote sensors; (D) setting joint weights of the effective kinect sensors of the selected effective kinect sensors on the basis of joints of the skeleton structure images; And (e) compositing a skeleton structure image of the selected valid Kinect sensors based on the set weights, wherein the weights include a first weight based on track state information of the valid Kinect sensors, And a second weight based on a direction to the valid Kinect sensors.

According to the present invention, there is an advantage that a user can stably obtain a skeleton structure image even if the direction of the user is changed by rotation or the like.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a view showing an arrangement of a Kinect sensor used in a user orientation estimation apparatus according to an embodiment of the present invention; FIG.
2 is a block diagram illustrating a structure of a user orientation estimation apparatus according to an embodiment of the present invention.
3 illustrates an operation for selecting an effective keynote using forward vector tracking according to an embodiment of the present invention;
4 is a diagram showing an example of selecting an effective kinect sensor according to one embodiment of the present invention;
5 is a view for explaining a method of setting a weight based on a camera direction according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating an overall flow of a method of estimating a user attitude according to an embodiment of the present invention. FIG.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing an arrangement structure of a Kinect sensor used in a user orientation estimation apparatus according to an embodiment of the present invention.

The present invention relates to a method and an apparatus for estimating a posture of a user by grasping a joint structure of a user stably against a user's posture rotation. 108 and 110 are used.

The Kinect sensors 100, 102, 104, 106, 108, and 110 are motion recognition cameras, and are sensors that recognize a user's body motion and play games and entertainment. The Kinect sensors 100, 102, 104, 106, 108, and 110 can track the motion of the captured person by exploring a camera capable of sensing depth through an infrared projector. Particularly, it is possible to acquire a skeleton structure representing the connection structure of the joints of a person captured using the Kinect sensor, and the skeleton structure acquired by the Kinect sensor can be used as basic information for estimating a human posture have.

Referring to FIG. 1, a user orientation estimation apparatus according to an embodiment of the present invention is arranged in a surround structure such that a plurality of Kinect sensors 100, 102, 104, 106, 108, and 110 surround the user as a whole.

According to one embodiment, the Kinect sensors 100, 102, 104, 106, 108, 110 may be arranged to be generally circular in 60 degree increments. Of course, those skilled in the art will appreciate that the number of Kinect sensors that may be deployed as needed can vary. Nine Kinect sensors may be arranged at intervals of 40 degrees, and four Kinect sensors may be arranged at intervals of 90 degrees.

Preferably, the user to be tracked may be located at the center of the Kinect sensors arranged to form a generally circular shape.

In the virtual reality entertainment system or the military virtual reality system, it is necessary to estimate the user's posture even when the user is rotating. The present invention uses a plurality of Kinect sensors to accurately estimate the user's posture regardless of the user's rotational state This structure allows some of the plurality of Kinect sensors to capture the front of the user regardless of the direction the user is facing.

Each of the plurality of Kinect sensors 100, 102, 104, 106, 108, and 110 independently captures the user's image and independently acquires an image of the skeleton structure.

2 is a block diagram illustrating a structure of a user orientation estimation apparatus according to an embodiment of the present invention.

2, a user orientation estimation apparatus according to an embodiment of the present invention includes a kinematic sensor unit 200, a common coordinate transformation unit 202, an effective kinetic sensor selection unit 204, a weight setting unit 206 A skeleton synthesis unit 208, a post-processing unit 210, and a posture estimation unit 212. [

The Kinect sensor unit 200 includes a plurality of Kinect sensors, and acquires a skeleton structure image of the target user as shown in FIG. The skeleton structure image is included in the three-dimensional coordinate system, and the connection state of many joints is expressed in the skeleton structure image. As described above, the plurality of Kinect sensors 100, 102, 104, 106, 108, and 110 constituting the Kinect sensor unit 200 independently acquire the skeleton structure image.

The common coordinate conversion unit 202 converts the skeleton structure image acquired by each of the kinect sensors into an image for the coordinates of the common coordinate system. The skeleton structure image acquired from each Kinect sensor is represented by the coordinate value of the coordinate system based on each Kinect sensor. Since the skeleton structure image obtained by each of the plurality of Kinect sensors is synthesized, it is necessary to express the skeleton structure image acquired by each Kinect sensor in the common coordinate system, and the common coordinate transformation unit 202 performs coordinate transformation .

The common coordinate transformation performed by the common coordinate transformation unit 202 can be expressed as Equation (1).

는 공통 좌표계로 변환된 좌표값이고,

은 각 키넥트 센서의 좌표계에 표현된 좌표값이다. 한편, R은 회전 매트릭스이며, 일례로 3 X 3의 디멘션을 가지는 매트릭스일 수 있다. T는 변형(Translation) 매트릭스이며, 일례로 3 X 1의 디멘션을 가지는 매트릭스일 수 있다. In Equation (1) above,

Is a coordinate value converted into a common coordinate system,

Is the coordinate value expressed in the coordinate system of each Kinect sensor. On the other hand, R is a rotation matrix, and may be, for example, a matrix having a dimension of 3 X 3. T is a translation matrix, e.g., a matrix having a dimension of 3 X 1.

A rotation matrix T and a transformation matrix T for transformation into a common coordinate system for each kinetic sensor are obtained, and R and T may be obtained by a predetermined calibration operation.

According to an embodiment of the present invention, a correction operation for calculating the rotation matrix R and the transformation matrix T using a check board can be performed. The correction using the check board uses the RGB camera in the same way as the conventional Kinect sensor.

On the check board, two types of squares can be arranged. By using the coordinate values in the common coordinate system corresponding to the vertices of each square on the check board and the coordinate values of the Kinect sensors, .

According to another embodiment of the present invention, a correction operation may be performed using an infrared marker. For implementation of the present invention, the skeleton structure image acquired by the Kinect sensor is output based on the coordinate system of the infrared camera of the Kinect sensor.

However, the correction process using the most common check board is performed using the RGB camera coordinate system of the Kinect sensor. Of course, since the RGB camera and the infrared camera are located adjacent to each other, the positions of the RGB camera and the infrared camera are similar, and the RGB camera coordinate system can be approximated to be used as the coordinate system of the infrared camera. In order to convert to a precise coordinate system, it is preferable to carry out the correction work using the IR camera. Since the existing check board is not accurately identified in the infrared image, the correction using the infrared marker is performed It might be. According to an embodiment of the present invention, infrared markers may be arranged at preset intervals, and the infrared markers disposed may be detected by using an infrared camera to perform a correction operation.

The effective kinect sensor selection unit 204 selects a Kinect sensor to be used for self-estimation among the plurality of Kinect sensors 100, 102, 104, 106, 108, and 110 arranged to surround the sensing target user Function. The Kinect sensor generates the user's skeleton structure image on the assumption that the user is looking at the Kinect sensor in front. However, as shown in FIG. 1, some of the plurality of Kinect sensors photograph the front of the user, while some of the Kinect sensors are disposed at the position of photographing the rear face of the user.

The effective keynote sensor selection unit 204 selects only the keynote sensor that captures the front of the user among the plurality of keynote sensors and excludes the keynote sensor that captures the back of the user from the selection. The skeleton structure image obtained from the excluded Kinect sensor is not used for the synthesis of the skeleton structure image in the future. This is because the skeleton structure image obtained from the excluded Kinect sensor is a skeleton structure image generated by photographing the back side of the user, and thus the accuracy of the skeleton structure image can not be relied upon since it is an inverted skeleton structure.

 4 is a diagram showing an example of selecting an effective kinect sensor according to an embodiment of the present invention.

As shown in FIG. 4, when the user is looking at the 12 o'clock direction, the keynote sensors 100, 102, and 104 located at the upper portion in FIG. 4 include a keynote sensor that captures the front of the user, The kinetic sensors 106, 108, and 110 located at the rear of the user are kinetic sensors that capture the back surface of the user. Therefore, the lower-positioned kinect sensors 106, 108, and 110 are not selected by the effective-knot sensor selection unit 204. [

The effective Kinect sensor selection unit 204 may select a Kinect sensor for photographing the front of the user in various ways. According to a preferred embodiment of the present invention, the forward vector is tracked to determine whether each kinetic sensor captures the front of the user and to select a valid Kinect sensor.

Here, the forward vector means a vector with respect to the direction in which the user is looking. 3 is a diagram for explaining an operation of selecting an effective keynote using front vector tracking according to an embodiment of the present invention.

According to an embodiment of the present invention, the forward vector can be defined as a vector orthogonal to the line connecting the left shoulder and the right shoulder, which is tracked using the left shoulder and the right shoulder in the skeleton structure image of the Kinect sensor.

The effective keynote sensor selection unit 204 first calculates the forward vector of the skeleton structure image synthesized in the previous frame for forward vector tracking in the current frame. In Fig. 3, the forward vector of the skeleton structure image synthesized in the previous frame is indicated by a dotted line.

Further, a forward vector for each of the kinetic sensors of the current frame is calculated from the skeleton structure image of each kinetic sensor in the current frame. In Fig. 3, the forward vectors of the respective keynote sensors in the current frame are indicated by solid lines.

Whether or not each key connector sensor is facing the front of the user is determined by the inner product of the forward vector of the skeleton guage image synthesized in the previous frame and the forward vector of each key sensor in the current frame.

If the inner product of the forward vector of the previous frame and the forward vector of the specific Kinect sensor in the current frame is positive, it is determined that the corresponding Kinect sensor is disposed toward the front of the user. 3 (a) shows a case in which the forward vector of the Kinect sensor in the current frame and the inner product of the forward vector of the skeleton structure image synthesized in the previous frame are positive numbers. In this case, the corresponding keynote sensor is selected by the valid keynote sensor selection unit 204.

Meanwhile, FIG. 3B shows a case where the inner product of the forward vector of the Kinect sensor in the current frame and the forward vector of the skeleton structure image synthesized in the previous frame is negative. In this case, the corresponding keynote sensor is not selected by the valid keynote sensor selection unit 204.

When the key knot sensor selection is made by the effective knot sensor selection unit 204, the weight setting unit 206 sets the weight for each joint of the selected knot sensors. According to a preferred embodiment of the present invention, the weight is set in consideration of two pieces of information. The first information is information on the track state of the Kinect sensor, and the first weight is set for each joint of the Kinect sensors by using the information. The second information is information on the camera direction, and a second weight is set using the information. The final weight is determined by the product of the first weight and the second weight.

Specifically, the first information provides the status information for each joint when the skeleton structure image is generated by the Kinect sensor. At this time, the joint state is divided into three states: tracked, estimated, and track-failed. A weight of 1 is given in case of track completion, a preset weight of less than 1 in case of estimation, and a weight of 0 is given to track failure.

The second weight based on the Kinect sensor orientation is specifically a weight of whether the Kinect sensor orientation is appropriate for a specific region of the joints of the left and right regions to photograph the joint region. Here, the joints in the two regions are the left area joint and the right area joint. For example, the left shoulder joint and the left wrist joint are included in the left area joint. The right shoulder joint and the right wrist joint are included in the right area joint. On the other hand, joints located on the front of the person, such as the jaw joints, do not correspond to the joints of the right region of the left region. Among the multiple joints obtained from the skeleton structure image, the left region joint and the right region joint are preset.

Whether or not it is appropriate to photograph the left area joint and whether it is appropriate to photograph the right area joint is determined based on the direction of the camera. If a particular Kinect sensor is not located within a predetermined left area, it is determined that the Kinect sensor is not suitable for photographing the left area joint, and if the specific Kinect sensor is not located within a preset right area, The sensor is judged to be inadequate for imaging right field joints.

, The right region and the left region are classified based on the forward vector. The forward vector can be determined based on the current frame or the previous frame, and defines the first angular region as the left region and the second angular region as the right region based on the forward vector. At this time, the left area and the right area may be set to have overlapping areas.

According to an embodiment of the present invention, when the forward vector is 0, the left region may define a 90-degree to + 30-degree region as a left region, and the right region may define a 30-degree to + can do.

If the direction of the specific Kinect sensor is not included in the left area from the user center for the second weight setting, the pre-set left joints of the skeleton structure image captured by the Kinect sensor may be preset to less than 1 And gives a second weight. In the same manner, when the direction of a specific Kinect sensor from the user center is not included in the right region, a predetermined second subordinate position for a predetermined right region joints among the joints of the skeleton structure image captured by the Kinect sensor Weight is assigned.

5 is a view for explaining a method of setting a weight based on a camera direction according to an embodiment of the present invention.

Referring to FIG. 5, the forward vector 500 is set to 0 degrees, and the left shoulder direction is 90 degrees from the center of the user and the right shoulder direction is +90 degrees.

In FIG. 5, the first to third Kinect sensors 100, 102, and 104 are the Kinect sensors selected by the effective Kinect sensor selection unit 204 because they photograph the front of the user, and the fourth to sixth Kinect sensors 100, The sensors 106, 108, and 110 are Kinect sensors that are not selected by the effective Kinect sensor selection unit 204 because they photograph the back of the user.

With respect to the -90 degrees to 30 degrees area of the left area, the first Kinect sensor 100 and the second Kinect sensor 102 have an angle from the center of the user to each Kinect sensor in the left area. A first weight is set to 1 for the first Kinect sensor 100 and the second Kinect sensor. However, the angle from the center of the user to the third Kinect sensor 104 is not in the left region, and for the left area joints acquired by the third Kinect sensor 104, a predetermined weight less than 1 is applied do. Of course, a weight of one is applied to the right region joints acquired by the third Kinect sensor 104. [

The weights of the respective knee sensors of the respective knee sensors are finally determined in a form in which the first weight and the second weight are multiplied.

The skeleton synthesis unit 208 synthesizes the skeleton structure images of the respective Kinect sensors selected by the effective kinect sensor selection unit 204 by applying the weights set by the weight setting unit. Since weights are set for each joint of each Kinect sensor, a general weighted average method can be used for image synthesis.

)을 나타낸 식이다. (2) is a skeleton structure image synthesis for each joint in the skeleton synthesis unit 208

).

In the above equation, P denotes a three-dimensional coordinate of the skeleton structure image, n denotes an index indicating a Kinect sensor, and i denotes an index representing a joint.

는 다음의 수학식 3과 같이 정의될 수 있다. On the other hand,

Can be defined as the following equation (3).

는 키넥트 센서의 추적 상태에 따른 제1 가중치를 의미하고,

는 키넥트 센서의 방향에 따른 제2 가중치를 의미한다. In the above equation (3)

Means a first weight according to the tracking state of the Kinect sensor,

Quot; means a second weight according to the direction of the Kinect sensor.

The post-processing unit 210 performs post-processing on the synthesized skeleton structure image. In the synthesized skeleton, the length of each joint may be slightly different from that of the actual user. The post-processing unit 210 performs a post-process by multiplying each preset joint correction coefficient by a predetermined correction coefficient to correct the difference.

The attitude estimation unit 212 estimates the attitude of the user based on the synthesized and post-processed skeleton structure image. Since the posture estimation based on the skeleton structure image is a general technique, a detailed description thereof will be omitted.

6 is a flowchart illustrating an overall flow of a method of estimating a user attitude according to an embodiment of the present invention.

Referring to FIG. 6, first, a plurality of Kinect sensors arranged to surround an object user acquires a user's skeleton structure image independently (step 600).

Each Kinect sensor converts the acquired skeleton structure image into coordinates of a common coordinate system (step 602). The rotation matrix R and the transformation matrix T for conversion to the common coordinate system are previously set for each of the kinetic sensors by a preset calibration operation.

A Kinect sensor for photographing the front of the user out of the plurality of Kinect sensors is selected as the effective Kinect sensor (step 604). Only the skeleton structure images of the Kinect sensors that are effectively selected are used for the synthesis of the skeleton structure image.

If an effective Kinect sensor is selected, a weight to be applied to the skeleton structure image for each joint of each Kinect sensor is set (step 606).

The weight is divided into a first weight according to the track state of the Kinect sensor and a second weight according to the direction of the Kinect sensor and the wearer. The final weight is set by multiplying the first weight and the second weight.

When the weight setting is completed, the weighted values are applied to synthesize the skeleton structure images obtained from the valid Kinect sensors (step 608).

After the skeleton structure image is synthesized, post-processing for correcting the length of each joint in the synthesized image is performed (step 610), and the posture of the user is estimated based on the post-processed skeleton structure image 612).

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- And various modifications and changes may be made thereto by those skilled in the art to which the present invention pertains. Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

Claims (15)

A plurality of Kinect sensors arranged to surround the user and the plurality of Kinect sensors independently acquiring the image of the user's skeleton structure;
A common coordinate system conversion unit for converting the user's skeleton structure image acquired by the plurality of Kinect sensors into a common coordinate system;
An effective keynote sensor selection unit for selecting keynote sensors for photographing the front of the user among the plurality of keynote sensors;
A weight setting unit for setting weights for the joints of the skeleton structure images of the respective effective knee pick sensors for the selected effective knee pick sensors; And
And a skeleton synthesis unit for synthesizing the skeleton structure images of the selected effective kinect sensors based on the set weights,
Wherein the weight includes a first weight based on track state information of the valid Kinect sensors and a second weight based on a direction from the user to the effective Kinect sensors.
The method according to claim 1,
Wherein the effective kinect sensor selection unit uses a forward vector orthogonal to a line connecting the left shoulder and the right finger of the user in a skeleton structure image captured by the kinect sensors.
3. The method of claim 2,
Wherein the effective kinect sensor selection unit determines whether to photograph the front of the user through an inner product of a front vector of a previous frame and a forward vector of each kinetic sensor in a current frame.
The method of claim 3,
Wherein the first weight is set for each joint based on state information about a joint of the skeleton structure image of each effective Kinect sensor.
The method according to claim 1,
Wherein the weight setting unit defines a left area and a right area based on the user's forward vector in the skeleton structure image of each valid Kinect sensor and determines a direction from the user to a specific Kinect sensor among the valid Kinect sensors Applying a predetermined second weight less than 1 to pre-established left region joints of the joints of the skeleton structure image captured by the corresponding Kinect sensor when not in the left region, A predetermined second weight less than 1 for pre-established left region joints of the joints of the skeleton structure image captured by the corresponding Kinect sensor when the direction of the sensor to the specific Kinect sensor is not in the right region Is applied to the user.
The method according to claim 1,
Wherein the skeleton synthesis unit applies a weight corresponding to a product of the first weight and the second weight to each joint and synthesizes the skeleton structure image by weighted averaging.
The method according to claim 1,
And an attitude estimation unit for estimating the attitude of the user based on the skeleton structure image synthesized by the skeleton synthesis unit.
A plurality of Kinect sensors arranged to surround the user and the plurality of Kinect sensors independently acquiring the image of the user's skeleton structure;
A common coordinate system conversion unit for converting the user's skeleton structure image acquired by the plurality of Kinect sensors into a common coordinate system;
An effective keynote sensor selection unit for selecting keynote sensors for photographing the front of the user among the plurality of keynote sensors;
A weight setting unit for setting weights for the joints of the skeleton structure images of the respective effective knee pick sensors for the selected effective knee pick sensors; And
And a skeleton synthesis unit for synthesizing the skeleton structure images of the selected effective kinect sensors based on the set weights,
The effective kinect sensor selection unit uses a forward vector orthogonal to a line connecting the left shoulder and the right finger of the user in a skeleton structure image captured by the kinect sensors, And determines whether to photograph the front of the user through an inner product of a forward vector for each of the kinetic sensors.
A method of estimating a user's attitude in an environment in which a plurality of Kinect sensors are arranged to surround a user,
(A) the plurality of Kinect sensors independently acquiring the user's skeleton structure image;
(B) transforming the user's skeleton structure image acquired from the plurality of Kinect sensors into a common coordinate system;
Selecting effective keynote sensors for capturing a front of the user among the plurality of keynote sensors;
(D) setting joint weights of the effective kinect sensors of the selected effective kinect sensors on the basis of joints of the skeleton structure images; And
(E) compositing a skeleton structure image of the selected effective Kinect sensors based on the set weights,
Wherein the weight comprises a first weight based on track state information of the valid Kinect sensors and a second weight based on a direction from the user to the effective Kinect sensors.
10. The method of claim 9,
Wherein the step (c) uses a forward vector orthogonal to the line connecting the left shoulder and the right finger of the user in the skeleton structure image captured by the Kinect sensors.
11. The method of claim 10,
Wherein the step (c) decides whether or not to photograph the front of the user through an inner product of a front vector of the previous frame and a forward vector of each kinetic sensor in the current frame.
12. The method of claim 11,
Wherein the first weight is set for each joint based on state information about a joint of the skeleton structure image of each effective Kinect sensor.
10. The method of claim 9,
Wherein the step (d) defines a left area and a right area based on the forward vector of the user in the skeleton structure image of the respective effective Kinect sensors, Applies a predetermined second weight less than 1 to predetermined left joints of the joints of the skeleton structure image captured by the corresponding Kinect sensor when the direction of the left knee sensor is not in the left region, If the direction to the specific Kinect sensor of the effective Kinect sensors is not within the right area, then a pre-set less than 1 preset for the left area joints of the skeleton structure image captured by the Kinect sensor And a second weight is applied to the user.
10. The method of claim 9,
Wherein the step (e) applies a weight corresponding to a product of the first weight and the second weight to each joint, and synthesizes the skeleton structure image by weighted averaging.
10. The method of claim 9,
And estimating the posture of the user based on the skeleton structure image synthesized by the step (e).

Images