KR102150172B1 - Relative movement based motion recognition method and apparatus - Google Patents

Abstract

The present invention discloses a motion recognition method and apparatus based on relative motion. According to the present invention, (a) obtaining a first rotation value according to movement from a first posture to a second posture from a plurality of sensors attached to a plurality of body parts-the first rotation value is based on a global coordinate system. It is the value rotated by -; (b) calculating a first rotation change amount from the first rotation value to the first posture and the second posture; (c) converting the first rotation value into a second rotation value based on a world coordinate system of software for modeling by using the first rotation change amount and the transformation matrix; (d) calculating a second rotation change amount of each of the modeling body parts by using a rotation value of each modeling body part used in the software and a rotation value of an upper body part of each body part; (e) obtaining a third rotation value of each of the modeling body parts based on the world coordinate system of the software by using the second rotation change amount; And (f) converting a rotation value obtained from the plurality of sensors using the second rotation value and the third rotation value into a rotation value applied to the modeling body part. A method is provided.

Description

Relative movement based motion recognition method and apparatus

The present invention relates to a motion recognition method and apparatus based on relative motion.

In the past, there have been many attempts to accurately reproduce real human movements in various fields such as military training, sports, and entertainment in addition to the game and movie industry, and nowadays it has been named as a technique called motion capture that applies motion data to 3D character modeling. .

There are mainly visual tracking methods based on vision such as Kinect and Leap Motion, non-visual tracking methods that attach sensors (markers) to the body, and robot-aided tracking methods with the help of machines.

Among them, the visual tracking method is greatly affected by light and the surrounding environment, and the marker-based method has a problem that requires a high cost and a large amount of computation.

In the sensor-based method, it is mainly used as an IMU sensor. However, when purchasing a sensor and software together, it is burdensome because an expensive cost is required.

Even if the simulation software is not purchased directly, a lot of time and effort is required to directly simulate even a simple motion because the user must understand the coordinate system of the sensor directly used to check the correct motion and the calculation of rotation in 3D space.

In addition to these drawbacks, when using existing software, limitations such as a fixed attachment direction of a sensor or a person's posture required in a measurement step increase inconvenience.

In order to solve the above-described problems of the prior art, the present invention is to propose a motion recognition method and apparatus based on relative motion in which even a user unfamiliar with motion capture can perform a simulation with only a simple setting.

In order to achieve the above object, according to an embodiment of the present invention, (a) a first rotation value according to a movement from a first posture to a second posture from a plurality of sensors attached to a plurality of body parts Acquiring-the first rotation value is a value rotated based on a global coordinate system -; (b) calculating a first rotation change amount from the first rotation value to the first posture and the second posture; (c) converting the first rotation value into a second rotation value based on a world coordinate system of software for modeling by using the first rotation change amount and the transformation matrix; (d) calculating a second rotation change amount of each of the modeling body parts by using a rotation value of each modeling body part used in the software and a rotation value of an upper body part of each body part; (e) obtaining a third rotation value of each of the modeling body parts based on the world coordinate system of the software by using the second rotation change amount; And (f) converting a rotation value obtained from the plurality of sensors using the second rotation value and the third rotation value into a rotation value applied to the modeling body part. A method is provided.

The first to third rotation values may be expressed as a quaternion including a rotation angle and a rotation axis vector.

Steps (a) to (f) are performed in the initial T posture, and at least some of the modeling body parts may have different coordinate systems.

The plurality of sensors may be IMU sensors.

The third rotation value may be a value in which each of the modeling body parts used in the software is rotated based on the world coordinate system.

The third rotation value may be obtained through successive multiplication of hierarchical upper body parts of each of the modeling body parts.

According to another aspect of the present invention, there is provided an apparatus for recognizing a motion based on a relative motion, comprising: a processor; And a memory connected to the processor, wherein the memory calculates a first rotation change amount using a first rotation value according to a movement from a first posture to a second posture from a plurality of sensors attached to a plurality of body parts. And-the first rotation value is a value rotated based on the global coordinate system -; The first rotation value is converted into a second rotation value based on the world coordinate system of the software for modeling using the first rotation change amount and the transformation matrix, and the rotation value of each modeling body part used in the software and the The second rotation change amount of each of the modeling body parts is calculated using the rotation value of the upper body part of each body part, and the second rotation change amount of each of the modeling body parts is calculated based on the world coordinate system of the software using the second rotation change amount. 3 Acquiring a rotation value, and converting the rotation values obtained from the plurality of sensors into rotation values applied to the modeling body part using the second rotation value and the third rotation value, executable by the processor A motion recognition apparatus based on relative motion for storing program instructions is provided.

According to the present invention, there is an advantage of being able to correspond to a coordinate system of a modeled body part from a sensor without calibration.

1 is a diagram illustrating a process of calculating a rotational change amount according to an embodiment of the present invention.
FIG. 2 is a diagram showing that the coordinate states of each body part are different for each modeling even though they have the same posture.
3 is a diagram showing the hierarchical division of body parts.
4 is a diagram illustrating a point in which each body part of modeling has a different coordinate system in an initial T posture.
5 is a diagram illustrating a configuration of a motion recognition apparatus based on relative motion according to an embodiment of the present invention.
6 is a view showing that the motion recognition method based on the relative motion according to the present embodiment is independent of the attachment state of the sensor and can be used equally in various modeling software.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar elements.

The present invention relates to an algorithm for automating a complex intermediate process in motion capture simulation, and to a method and an apparatus capable of applying sensor data to modeling for simulation without complicated calibration.

Hereinafter, the process of sensor data, rotation order, simulation program, character modeling, and coordinate system conversion will be sequentially described.

The IMU sensor used for motion capture consists of a 3-axis accelerometer and an angular velocity meter, and may include a geomagnetic sensor.

Based on the Earth's gravity and magnetic field, the data output from the three sensors is corrected with a filter to estimate the rotational state of the object.

The direction and rotation method of the 3-axis coordinate system as a reference for each sensor are defined in advance, and the sensor used in this embodiment (Xsens MTw Awinda) uses the right-speed coordinate system.

In addition, since the direction of the earth's magnetic field can be estimated from the geomagnetic sensor, a fixed coordinate system is defined and the difference in the relative rotation value with the sensor coordinate system is output as a result.

The format of rotation data to obtain the result value mainly uses Euler Angles, Quaternion, and Rotation Matrix, which can be converted to each other through mathematical formulas.

The Euler angle is not suitable for accurate rotation calculation due to a widely known problem called Gimbal Lock, and thus rotation calculation through a quaternion format is used in this embodiment.

이며, 이는 다시

로 표현이 가능하다. If we get the data of the rotation value relative to the global coordinate of the sensor in quaternion

Is, which again

It can be expressed as

(Sensor to Grobal)은 센서가 글로벌 좌표계를 기준으로 회전한 정도를 나타내는 상대적인 회전 값으로 센서로부터 출력되는 데이터이다. here,

(Sensor to Grobal) is a relative rotation value representing the degree of rotation of the sensor based on the global coordinate system, and is data output from the sensor.

는 회전 각이고, v=Vector3(

)는 회전 축 벡터이다. Also, w=

Is the rotation angle, v=Vector3(

) Is the rotation axis vector.

,

)을 획득하고, 제1 회전 값을 통해 제1 자세와 제2 자세로의 제1 회전 변화량(

)을 계산한다. According to an embodiment of the present invention, as shown in FIG. 1, a rotation value according to a movement from a first posture (0) to a second posture (1) from a plurality of body-attached sensors (first rotation value ,

,

) Is obtained, and the amount of change in the first rotation to the first posture and the second posture (

) Is calculated.

In a three-dimensional space, it is divided into a left-handed coordinate system or a right-handed coordinate system depending on whether the rotation direction is clockwise or counterclockwise. In addition, it is divided into several combinations according to the order of the rotating axis. In Euler angles, 12 types of X, Y, and Z rotation orders are classified into'Proper Euler angles' and'Tait-Bryan angles'.

In the Tait-Bryan method, each rotation based on the X, Y, and Z axes is also expressed as Roll, Pitch, and Yaw, and the conversion calculation formula for the quaternion and rotation matrix based on the rotation order of Euler angles is also different. You should use the formula.

The sensor used in this embodiment rotates in a counterclockwise direction according to the XYZ order, and the equation for converting the rotation data obtained in the form of quaternions back to the Euler angle is as follows.

Software such as Unreal, Unity, Blender, OpenGL, etc. that maps the data to 3D modeling, or a self-developed program, is used to check whether the motion is properly implemented using the data obtained from the sensor.

At this time, since each software has a different coordinate system, and the modeling used is also affected by other body parts connected to each body part, the coordinate system may be different, so finally, it must be accurately calculated by considering the three types of coordinate systems: sensor, software, and modeling. You can check the correct behavior of the data.

In the case of the unity used in this embodiment, a left-handed coordinate system with the Y-axis upward, the Z-axis toward the front, and the X-axis toward the right is used, and the rotation direction is also rotated in a clockwise direction opposite to the sensor.

The rotation order is also different from the sensor used in this embodiment in the ZXY order.

When the modeling to which the data is applied is loaded, it can be seen that several body parts are hierarchically connected and each has a different coordinate system state.

This is because different body parts are connected in the middle of each model to be used and the accumulated rotation value is different. As shown in FIG. 2, even though the body parts are in the same posture, the coordinate state of each body part may be different for each modeling.

For example, in the case of leg movement, when the thigh is moved, the calves and feet connected to the substructure are also affected by rotation.At this time, although the state of the local coordinate system of each body part does not change, the rotation value viewed based on the global coordinate system is different. Because it loses, it is necessary to consider the rotation values of the upper body part connected to each body part.

Therefore, since the calculation becomes more complicated as the number of connected body parts in the upper part increases, research has been conducted to simplify some body parts involved in major movements according to the structure of the human body instead of tracking all bone parts.

In this embodiment, body parts frequently used for modeling and animation are divided in stages in the form of FIG. 3.

For example, the calf area is defined as being affected by the upper 2 levels of the whole body and the thigh, and the foot area as being affected by the top 3 levels of the whole body, the thigh, and the calf.

After obtaining the data of the body part measured from the sensor, in order to implement the desired motion by connecting to each body part in the modeling, the correct direction to rotate for the next motion in the posture must be calculated.

To this end, the first rotation value is converted into a second rotation value based on the world coordinate system of the software for modeling by using the first rotation change amount and the transformation matrix obtained from the sensor. This can be expressed as an equation as follows.

(Global to World)는 센서의 글로벌 좌표계를 소프트웨어의 월드 좌표계로 변환시키는 변환 매트릭스이다. here,

(Global to World) is a transformation matrix that converts the sensor's global coordinate system to the software's world coordinate system.

In addition, in the point of using the Unity software in this embodiment, the vector part is changed to the software World coordinate system by multiplying the transformation matrix, and the angle part is changed from counterclockwise rotation to clockwise rotation, so a negative sign (-) is added.

Finally, it is necessary to consider the difference in rotation values for each body part in modeling.

As shown in FIG. 4, in the initial T posture, each body part of the modeling has a different coordinate system.

라 할 때,

(제2 회전 변화량)를 계산해야 한다. The rotation value of each body part

When la,

(2nd rotation change) must be calculated.

That is, the second rotational change amount of each modeling body part is calculated using the rotation value of each modeling body part used in the software and the rotation value of the upper body part of each body part.

는 다음과 같이, 모델링 신체 부위 각각의 계층적 상위 부위들의 연속적인 곱셈을 통해 획득된다.

Is obtained through successive multiplication of hierarchical upper parts of each modeling body part as follows.

는 다음과 같이 계산된다. For example, of Right Lower Arm 5 in Figure 4

Is calculated as follows.

)이 획득된다.As described above, the third rotation value of each of the modeled body parts based on the world coordinate system of the software (

) Is obtained.

(World to Part)는 소프트웨어에서 사용되는 모델링 신체부위가 World 좌표계를 기준으로 어느 정도 회전했는지 나타내는 값으로서, 상기한 바와 같이, 상위 신체 부위들의 연속적인 곱셈을 통해 얻어진다. here,

(World to Part) is a value indicating how much the modeling body part used in the software has rotated based on the World coordinate system, and is obtained through successive multiplication of upper body parts as described above.

For example, the calf area requires 3 quaternion multiplication in the order of calf, thigh, and full body, and when tracking the foot of the lower body area, 4 multiplications are required in the order of foot, calf, thigh, and whole body.

와 상기한

와 곱하면 센서로부터 모델링 신체 부위의 좌표계에 대응하는

를 얻을 수 있다. Obtained through the above process

And above

Multiplied by and corresponds to the coordinate system of the modeled body part from the sensor

Can be obtained.

5 is a diagram illustrating a configuration of a motion recognition apparatus based on relative motion according to an embodiment of the present invention.

As shown in FIG. 5, the device according to the present embodiment may include a processor 500 and a memory 502.

The processor 500 may include a central processing unit (CPU) capable of executing a computer program or a virtual machine.

The memory 502 may include a nonvolatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, or the like. The memory 502 may also include volatile memories such as various random access memories.

Program instructions executable by the processor 500 are stored in the memory 502.

In order to obtain a rotation value corresponding to the coordinate system of the modeled body part from the sensor without calibration, the memory 502 according to the present embodiment includes a movement from a first posture to a second posture from a plurality of sensors attached to a plurality of body parts. Obtaining a first rotation value according to-The first rotation value is a value rotated based on a global coordinate system-, calculates a first rotation change amount from the first rotation value to the first posture and the second posture, 1 Convert the first rotation value to a second rotation value based on the world coordinate system of the software for modeling using the rotation change amount and the transformation matrix, and the rotation value of each modeling body part used in the software and each of the above body parts. The second rotation change amount of each of the modeling body parts is calculated using the rotation value of the upper body part of, and the third rotation value of each modeling body part is calculated based on the world coordinate system of the software using the second rotation change amount. Program instructions for obtaining and converting the rotation values obtained from the plurality of sensors into rotation values applied to the modeling body part by using the second rotation value and the third rotation value are stored.

According to the present invention, as shown in FIG. 6, it is independent of the attachment state of the sensor, and various modeling software can be equally used.

The above-described embodiments of the present invention are disclosed for the purpose of illustration, and those skilled in the art having a general knowledge of the present invention will be able to make various modifications, changes, additions within the spirit and scope of the present invention, such modifications, changes and The addition should be viewed as falling within the scope of the claims.

Claims (10)

A method of performing motion recognition in a device including a processor and a memory connected to the processor,
(a) obtaining a first rotation value according to a movement from a first posture to a second posture from a plurality of sensors attached to a plurality of body parts-the first rotation value is a value rotated based on a global coordinate system- ;
(b) calculating a first rotation change amount from the first rotation value to the first posture and the second posture;
(c) converting the first rotation value into a second rotation value based on a world coordinate system of software for modeling by using the first rotation change amount and the transformation matrix;
(d) calculating a second rotational change amount of each of the modeling body parts using a rotation value of each modeling body part used in the software and a rotation value of an upper body part of each body part;
(e) obtaining a third rotation value of each of the modeling body parts based on the world coordinate system of the software by using the second rotation change amount; And
(f) converting rotation values obtained from the plurality of sensors into rotation values applied to the modeling body part using the second rotation value and the third rotation value,
Steps (a) to (f) are performed in an initial T posture, and at least some of the modeling body parts have different coordinate systems. The method of claim 1,
The first to third rotation values are a motion recognition method based on a relative motion expressed as a quaternion including a rotation angle and a rotation axis vector. delete The method of claim 1,
The plurality of sensors is an IMU sensor, a motion recognition method based on relative motion. The method of claim 1,
The third rotation value is a motion recognition method based on a relative motion in which each of the modeling body parts used in the software is rotated based on the world coordinate system. The method of claim 5,
The third rotation value is a motion recognition method based on a relative motion obtained through continuous multiplication of hierarchical upper body parts of each of the modeling body parts. As a motion recognition device,
Processor; And
Including a memory connected to the processor,
The memory,
(a) A first rotation change amount is calculated using a first rotation value according to a movement from a first posture to a second posture from a plurality of sensors attached to a plurality of body parts, and the first rotation value is a global coordinate system. It is a value rotated based on -;
(b) converting the first rotation value into a second rotation value based on the world coordinate system of the software for modeling using the first rotation change amount and the transformation matrix,
(c) calculating a second rotational change amount of each of the modeling body parts using a rotation value of each modeling body part used in the software and a rotation value of an upper body part of each body part,
(d) obtaining a third rotation value of each of the modeling body parts based on the world coordinate system of the software by using the second rotation change amount,
(e) to convert the rotation values obtained from the plurality of sensors into rotation values applied to the modeling body part by using the second rotation value and the third rotation value,
Store program instructions executable by the processor,
The (a) to (e) processes are performed in an initial T posture, and at least some of the modeling body parts have a motion recognition apparatus based on a relative motion having different coordinate systems. The method of claim 7,
The motion recognition apparatus based on relative motion, wherein the first to third rotation values are expressed as a quaternion including a rotation angle and a rotation axis vector. The method of claim 7,
The third rotation value is a motion recognition apparatus based on a relative motion, which is a value in which each of the modeling body parts used in the software is rotated based on the world coordinate system. The method of claim 9,
The third rotation value is a motion recognition apparatus based on a relative motion obtained through successive multiplication of hierarchical upper body parts of each of the modeling body parts.

Images