KR102648471B1 - Hybrid Motion Capture Position Switching Method for Virtual Fire Drill - Google Patents

Abstract

The present invention is a hybrid motion capture system that matches the size and ratio between the actual human body size and the human body model, and when optical position tracking fails, switches to motion sensor-based position tracking to continuously track the position of the human body. This is about how to switch the capture location.
The hybrid motion capture position switching method according to the present invention captures the position movement by photographing a marker attached to a rigid body worn on the human head through optical position measurement, and uses sensor-type motion capture to detect a plurality of motion sensors attached to the human body. In a hybrid motion capture method that analyzes signals to determine the movement of the human body and fuses the identified human body position information and movement information to implement a human body model in virtual space, the Z-axis position of the rigid body identified through the marker correcting the size and ratio of the human body model to the actual human body size using information; If the position of the human body cannot be tracked through optical position tracking, the position of the human body model is tracked through lower limb joint walking using motion sensor data acquired through sensor-type motion capture through positioning conversion. It includes a step of implementing movement, and is provided to provide uninterrupted continuity in tracking the human body.

Description

Hybrid Motion Capture Position Switching Method for Virtual Fire Drill}

The present invention relates to a hybrid motion capture position switching method. More specifically, in a hybrid motion capture system, the size and ratio between the actual human body size and the human body model are matched, and when optical position tracking fails, motion sensor-based position tracking is used. It relates to a hybrid motion capture position switching method that allows continuous tracking of the position of the human body by switching.

Firefighting officials who perform their duties in an increasingly complex and diverse disaster environment are increasingly in need of more intensive education and training. Since firefighters are currently unable to experience all dangerous disaster situations, they are improving their on-site response capabilities through various safety accident cases and tactical training.

However, these traditional training methods are limited by location and time, making it difficult to achieve effective training. Accordingly, training techniques using virtual reality technology have been emerging in recent years. Virtual training techniques have recently been in the spotlight because they are safe and can recreate real situations to give trainees a feeling of being in the real field.

To provide such virtual training, motion capture technology is used. Motion capture technology attaches sensors to objects such as people, animals, or machines to reproduce the movement information of the object, and is used in a variety of fields from the content field to industry and sports fields. there is. In addition, a hybrid motion capture method that measures dynamic signals related to rotation and movement by combining the positioning method of multiple cameras installed in an indoor space and the sensor-type motion reproduction method has been recently used. This hybrid motion capture system combines an optical position tracking system and sensor-type motion capture equipment to enable motion capture of multiple users in a large space.

However, in the process of fusing position and posture information through HMD (Head Mounted Display) or optical position tracking equipment with human body motion data using motion sensors, errors occur in the size difference between the actual trainee's body size and the virtual character. , Due to these errors, there was a problem in that unnatural phenomena occurred in the movement of the virtual character, such as feet slipping when floating in the air or walking.

In addition, when a trainee leaves the detection area of the optical position tracking equipment, it is difficult to track the trainee's location using only motion capture equipment, so the trainee disappears or remains stationary in the virtual space, making it difficult for training to proceed smoothly.

Republic of Korea Patent Publication No. 10-1768958 (registered on August 10, 2017) Republic of Korea Patent Publication No. 10-2188480 (registered on December 2, 2020)

The present invention was proposed to solve problems occurring in conventional virtual training systems. The purpose of the present invention is to match the size and ratio of the virtual human body model according to the size of the actual human body in the hybrid motion capture process and to match the size and ratio of the virtual human body model to the user in the virtual space. The goal is to provide a hybrid motion capture position switching method that allows continuous tracking of the human body by switching to a sensor-based tracking method if optical position tracking fails during the position tracking process.

In order to achieve the above object, the hybrid motion capture position switching method according to the present invention captures the position movement by photographing a marker attached to a rigid body worn on the human head through optical position measurement, and captures the position of the human body through sensor-type motion capture. A hybrid motion capture method that analyzes signals from a plurality of attached motion sensors to determine the movement of the human body and fuses the location information and movement information of the human body to create a human body model in virtual space, identified through the marker. correcting the size and ratio of the human body model to the actual human body size using the Z-axis position information of the rigid body; If the position of the human body cannot be tracked through optical position tracking, the position of the human body model is tracked through lower limb joint walking using motion sensor data acquired through sensor-type motion capture through positioning conversion. Includes a step of implementing movement.

Here, it is desirable to correct the size and ratio of the human body model in the virtual space according to the height of the human body by increasing the ratio so that the human body's feet touch the floor through T-Pose correction.

In addition, in order to eliminate the fall of the human body model that occurs when the position of the human body model according to optical position tracking is changed to the position of the human body model using motion sensor data when switching the positioning, the position of the head is calculated from the lowest human body object. This displaces the position of the human body model.

At this time, when the human body model is displaced using the motion sensor data, it is desirable to place a limit on the positional displacement to prevent the human body model from momentarily shaking.

Meanwhile, the tracking of the positional movement of the human body through the lower limb joint walking is performed by finding the ankle position in the human body and conversely moving the waist position based on the ankle through the positional displacement of the ankle joint. The movement of the human body according to the movement of the waist position is implemented and the movement of the position is tracked.

Here, the tracking of the positional movement of the human body through the lower limb joint walking is performed by calculating the position coordinates of the ankle connected to the joint angle from the waist of the human body, and between the position coordinates of the waist (Hip) and the ankle (Ankle). The direction vector of is obtained, and the movement of the human body is tracked by calculating the displacement of the waist relative to the ankle in each frame through the direction vector between the position coordinates of the hip and ankle.

At this time, calculate the displacement of the waist based on the ankle, compare the Z-axis positions of the two ankle joints, determine that the lowest ankle is connected to the floor, and move the waist based on the foot currently attached to the floor. And, when the foot on the floor switches, it is desirable to track the walking by moving the position of the waist based on the opposite foot.

The hybrid motion capture position switching method according to the present invention corrects and matches the size and ratio of the virtual human body model according to the size of the actual human body during the hybrid motion capture process, so that the virtual human body model can move naturally according to the movement of the human body. It works. In addition, if optical position tracking fails in the process of tracking the position of the human body in virtual space, it switches to a sensor-based tracking method according to the lower extremity joint gait and continues to track the human body, which has the effect of giving continuous continuity to human body tracking. there is.

1 is a conceptual diagram of an applied hybrid motion capture system;
Figure 2 is an example of installation of a motion sensor according to the present invention;
3 is a conceptual diagram of a hybrid motion capture algorithm according to the present invention;
Figure 4 is an example of the size of a 3D human body model being adjusted through T-Pose correction according to the present invention,
5 is a conceptual diagram of the lower limb joint walking algorithm according to the present invention;
Figure 6 is an example showing the positional relationship between the waist and ankles while walking according to the present invention;
Figure 7 is a flowchart showing the hybrid motion capture position switching process according to the present invention;
Figure 8 is an example of the position movement path of the human body during the hybrid motion capture position change process according to the present invention;
Figure 9 is an example of the fall of the human body model that occurs during positioning conversion according to the present invention;
Figure 10 is an example of the processing result after changing the positioning of the human body model according to the present invention;
Figure 11 is an example of an experimental environment for testing the performance of the hybrid motion capture position switching method according to the present invention;
Figures 12 and 13 show an example of the results of an experiment to determine whether users match when a user moves in room 1 according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 shows a conceptual diagram of a hybrid motion capture system applied to the present invention, and the present invention includes the contents of Patent No. 10-1768958 registered by the present applicant.

As shown in Figure 1, the hybrid motion capture system applied to the present invention analyzes the signal of a motion sensor attached to equipment worn by the human body through a sensor-type motion capture (Motionsensor Mocap system) to capture the three-dimensional movement (motion) of the human body. ), and through an optical position measurement (Optical position system), the marker attached to the equipment worn by the human body is photographed and analyzed to determine the positional movement, and the 3D movement information of the human body and the positional movement information of the human body are Through fusion, precise motion capture is performed. In addition, the hybrid motion capture system displays virtual human body model data that combines 3D movement information and positional movement information of the human body on the screen in real time through a monitoring program, and can be used with programs such as Motion Builder through plug-ins. Provided so that it can be linked.

Meanwhile, in a hybrid motion capture system, various equipment such as HMD (Head Mounted Display), gloves, and shoes can be used as motion sensors worn by the user to identify human body movements. Figure 2 shows an example of the installation of a motion sensor according to an embodiment of the present invention. In the embodiment of the present invention, a plurality of motion sensors (S) are attached to the joints of the human body to identify the movement of the human body. Each motion sensor (S) is given a unique ID to distinguish which joint the signal was measured from.

In addition, a vision marker (M) that can be recognized by a camera was attached to the head of the human body and photographed with an optical camera (C) to determine the movement of the marker (M). In an embodiment of the present invention, the marker (M) is attached to the motion sensor (S) attached to the head of the human body to provide a reference point for movement of the human body.

Figure 3 shows a conceptual diagram of a hybrid motion capture algorithm according to an embodiment of the present invention.

As shown in Figure 3, the hybrid motion capture system according to the present invention is a fusion of position data acquired through optical position measurement and full body motion sensor data acquired through sensor-type motion capture, and combines optical position measurement and sensor-type motion sensor data. By combining the advantages of motion capture, the user's location and movement can be tracked.

At this time, the hybrid motion capture algorithm uses a motion sensor to capture human body motion based on the hip. Since this motion data has no movement information, optical position tracking is applied to the head of the motion data. It allows people to move in three-dimensional space.

In addition, T-Pose Calibration is performed to increase the accuracy of position estimation of the rigid-body attached to the human body. This T-Pose calibration changes the size of the human body model in virtual reality to the Z of the rigid body. Correction is made using axis position information. If the head of the human body model is moved from the rigid body to the tracked position without data processing for the human body model, the feet appear to be under the ground or raised above the ground due to the height difference between the person and the human body model. Accordingly, in the present invention, when correcting the T-Pose, the proportions of the human body are increased to reach the floor, so that it can be readjusted into a virtual human body model similar to the height of an actual person. In other words, T-Pose correction refers to the process of matching the size and ratio of the human body model to the human body using the Z-axis information of the rigid body in a posture in which the human body is upright and arms open (T posture). Figure 4 shows an example in which the size of the 3D human body model is adjusted through such T-Pose correction.

On the other hand, if the human body movement is reproduced using only motion sensor data, there are limitations in reproducing the walking motion with the position of the waist fixed because the human body model cannot be moved. In the present invention, an algorithm for walking motion is implemented by analyzing the walking motion of the lower extremity joints of the human body in order to be able to track human body movement using motion sensor data even when optical position tracking is not possible.

That is, a lower limb joint walking algorithm using the ankle joint point was implemented by simplifying the human lower limb joints. Figure 5 shows a conceptual diagram of the lower limb joint walking algorithm according to an embodiment of the present invention.

In the in-place motion capture performed in sensor-type motion capture, the position of the waist (hip) is fixed, and other parts can freely express movement due to the motion sensor, but the lower limb joint walking algorithm first starts from the waist (hip) and joint angles. The direction vector between the calculated ankle position coordinates is obtained. Figure 6 shows the positional relationship between lower extremity joints, that is, the waist and ankles, during walking.

The difference between the calculated direction vector of the current frame and the previous frame means the amount of displacement of the ankle based on the hip position. If you think of this inversely, the ankle is fixed and the waist is moved based on the fixed ankle. becomes the displacement. Therefore, the movement of the human body is realized and tracked by calculating the displacement of the waist based on the ankle in each frame. At this time, by comparing the Z-axis positions of the two ankle joints, it is determined that the lowermost ankle is connected to the floor, and the waist is moved based on the foot currently attached to the floor. When the foot attached to the floor is switched, the waist is moved based on the opposite foot. By moving the position, the human body walks.

As such, in the present invention, when optical position tracking of the human body is not possible, the lowest ankle position is found to reconstruct the skeletal structure according to the lower limb joint walking algorithm, and human body movement is realized by moving the waist position through the position displacement of the ankle joint. This allows location tracking.

Below, the process of hybrid motion capture position switching using the lower limb joint walking algorithm of the present invention will be described.

Figure 7 is a flowchart showing a hybrid motion capture position switching process according to an embodiment of the present invention, and Figure 8 shows an example of a position movement path of the human body in the hybrid motion capture position switching process.

Step S100: First, when optical position tracking and sensor-type motion capture are performed smoothly, the hybrid motion capture system positions the human body by fusing position data acquired through optical position measurement and motion sensor data acquired through sensor-type motion capture. and movement are tracked and applied to the human body model whose size and ratio have been adjusted through T-Pose correction to realize the position and movement of the human body model in virtual space.

Steps S110, S120: While tracking the position and movement of the human body through hybrid motion capture, if the human body leaves the optical camera area or is blocked by other objects and optical position tracking fails (S110), the hybrid motion capture system Positioning is converted to human body position tracking through motion sensor data acquired through sensor-type motion capture (S120).

Step S130: On the other hand, even if the ratio between the human body and the model is adjusted through T-Pose correction, the difference between the position of the human body model according to optical position tracking and the position of the human body model using motion sensor data causes an instantaneous change in the two position information. This may cause the human body model to tremble up and down. In other words, during optical position tracking, a height error (fall) may occur between a human body model using head position information and a human body model using motion sensor data due to differences in human body proportions. Figure 9 shows an example of the fall of the human body model that occurs during such a positioning change. In the present invention, in order to eliminate this fall, when changing the positioning through motion sensor data, the position of the head is inverted from the lowest human body object to position the human body model. will change. At this time, the positional displacement of the model is limited to prevent the frame from shaking. Figure 10 shows an example of the processing results after changing the positioning of the human body model.

Step S140: When the position of the human body model is adjusted through the above process, the hybrid motion capture system tracks the position of the human body using motion sensor data acquired by sensor-type motion capture according to the lower limb joint walking algorithm.

Steps S150, S160: While the position of the human body is being tracked according to the lower extremity observation walking algorithm, when the user returns to the area of the optical camera (S150), the user returns to the optical position tracking method to track the position of the human body. The process of tracking human body position and movement through this hybrid motion capture system is repeated until the system is terminated (S160).

Below, we will verify the performance of the hybrid motion capture position switching method according to the present invention.

Figure 11 shows an example of an experimental environment for testing the performance of the hybrid motion capture position switching method. As shown in Figure 11, eight cameras were individually installed in two spaces to test hybrid motion capture position switching when passing through the blind spot of an optical camera, and for the experiment, a hard hat with an active marker attached and a motion sensor were installed. The experimental environment was configured so that the attached user could move around the experimental space. When a user is placed in an experimental environment, it is set up so that there is one-to-one matching with the virtual character (human body model).

Figures 12 and 13 are examples of the results of an experiment on whether users match when the user moves in Room 1. In Figure 12, when the optical camera did not recognize the safety helmet worn by the user, an experiment was conducted to move in virtual space by switching to a lower limb joint walking algorithm using a motion sensor according to the hybrid motion capture position switching method. This situation is outside the camera range, and it can be confirmed that the positioning is switched to motion sensor data-based positioning without difficulty. In addition, it can be confirmed that the experimenter is moving within the camera recognition range of Room 1 through the camera as a preparation process for moving to Room 2. Additionally, in Figure 13, it can be seen that the blind area beyond the room 1 camera recognition range is being converted to a motion data-based position without difficulty.

As such, the hybrid motion capture position switching method according to the present invention uses an optical method to track the movement of the human body in areas where optical position tracking is detected, and when optical position tracking is not detected, the human body is used through a lower limb joint walking algorithm. The movement of the device is tracked, allowing seamless motion capture not only inside the motion capture space but also when leaving and entering the space for a short period of time.

The present invention is not limited to the above-described embodiments, and various modifications and variations can be made by those skilled in the art to which the present invention pertains, within the scope of equivalency of the technical idea of the present invention and the scope of the claims described below. Of course, this can be achieved.

Claims (7)

Through optical position measurement, position movement is identified by photographing a marker attached to a rigid body worn on the human head, and through sensor-type motion capture, signals from multiple motion sensors attached to the human body are analyzed to determine the movement of the human body. , In a hybrid motion capture method that implements a human body model in virtual space by fusing the known position information and movement information of the human body,
(a) correcting the size and ratio of the human body model to the actual human body size using the Z-axis position information of the rigid body identified through the marker; (b) If the position of the human body cannot be tracked through optical position tracking, the position movement of the human body is tracked through lower limb joint walking using motion sensor data acquired through sensor-type motion capture through positioning conversion. It includes implementing a position movement of the model,
In step (a), the size and ratio of the human body model in virtual space are corrected by the height of the human body by increasing the ratio so that the human body's feet touch the floor through T-Pose correction,
In step (b), in order to eliminate the fall of the human body model that occurs when the position of the human body model according to optical position tracking is switched to the position of the human body model using motion sensor data when switching the positioning, the head is moved from the lowest human body object. The position of the human body model is displaced by inverting the position of the human body model, but a limit is placed on the positional displacement to prevent the human body model from momentarily shaking when the human body model's position is displaced.
The tracking of the positional movement of the human body through the lower limb joint walking is performed by finding the ankle position in the human body and conversely moving the waist position based on the ankle through the positional displacement of the ankle joint. It tracks the position movement by implementing the movement of the human body according to the movement, and calculates the position coordinates of the ankle connected to the joint angle from the human body's hip to determine the direction vector between the position coordinates of the hip and ankle. A hybrid motion capture position switching method characterized by tracking the movement of the human body by calculating the displacement of the waist based on the ankle every frame through the direction vector between the position coordinates of the hip and ankle. .
delete delete delete delete delete According to clause 1,
Calculate the displacement of the waist based on the ankle,
By comparing the Z-axis positions of the two ankle joints, it is determined that the lowest ankle is connected to the floor, and the waist is moved based on the foot currently connected to the floor. When the foot connected to the floor is switched, the waist is positioned based on the opposite foot. A hybrid motion capture position switching method characterized by tracking walking by moving .