KR20230122557A - Motion capture system compensation method using ground reaction force data - Google Patents

Abstract

The embodiment includes determining a ground contact area using pressure data received from a sensor; comparing a key and a value of the ground contact portion with a reference key and value of previously stored data; and storing a key and value for the ground contact portion in the previously stored data or extracting error data according to a comparison result.

Description

Motion capture system compensation method using ground reaction force data}

An embodiment of the present invention is a motion capture system correction method through ground reaction force data that performs precise motion capture by correcting the value of an ankle sensor more accurately by reducing the error through additional equipment (e.g., a ground reaction force measurement sensor), a foot pressure sensor). it's about

In the case of sensor-based motion capture, the error of the geomagnetic sensor due to the drift of the gyro sensor or the distortion of the magnetic field affects the estimation of the rotation value of the joint, resulting in joint distortion or abnormal rotation.

In particular, the sensor closer to the floor has a problem of generating a larger error than the sensor at other parts depending on the situation of the magnetic field caused by the wire passing through the floor of the measurement place.

According to an embodiment of the present invention, when applied to an existing gyro sensor-based motion recognition system, motion capture system correction through ground reaction force data that can easily correct posture without a motion alignment process through TPose, only for drift of the ankle sensor method can be provided.

The problem to be solved in the embodiment is not limited thereto, and it will be said that the solution to the problem described below or the purpose or effect that can be grasped from the embodiment is also included.

A motion capture correction method according to an embodiment may include determining a ground contact area using pressure data received from a sensor; comparing a key and a value of the ground contact portion with a reference key and value of previously stored data; and storing a key and value for the ground contact portion in the previously stored data or extracting error data according to a comparison result.

The key for the ground contact portion may include a joint angle received from motion data and a pressure value of pressure data for the ground contact portion.

The joint angles received from the motion data may be angles of joints other than the ankle.

The value of the ground contact portion may include an angle of the ankle in contact with the ground.

The extraction step is

It may include determining whether a key for the ground contact portion and a reference key of the previously stored data are the same.

If the key for the ground contact portion and the reference key of the pre-stored data are the same, the error in which the value for the ground contact portion and the reference value of the pre-stored data are reflected It may include; determining the data.

The error data is

Difference between the reference value of the previously stored data corresponding to the reference key of the pre-stored data and the value for the ground contact portion corresponding to the key for the ground contact portion can

It may include; reflecting the extracted error data to the angle of the ankle.

If the key for the ground contact part and the reference key of the pre-stored data are different, the key and value for the ground contact part are set to the reference key of the pre-stored data ) and adding to a reference value.

In the determining step, the ground contact portion includes one of both feet, one foot, and the air, and the comparing step may be performed when the ground contact portion in the determining step is both feet or one foot.

A motion capture and calibration device according to an embodiment includes a receiver configured to receive pressure data and motion data from a sensor; and a control unit, wherein the control unit determines a ground contact area using the pressure data received from the sensor, and sets a key and value for the ground contact area as a reference key of the pre-stored data. and a reference value, and according to the comparison result, a key and a value for the ground contact portion are stored in the previously stored data or error data is extracted.

According to the embodiment, in the case of the TPose process for correcting posture in general, there is inconvenience that the motion demonstrator has to stop the existing motion, but only for the drift of the ankle sensor, accurate 3D visualization is possible without the motion alignment process through TPose We implement a motion capture correction method through ground reaction force data.

Various advantageous advantages and effects of the present invention are not limited to the above description, and will be more easily understood in the process of describing specific embodiments of the present invention.

1 is a block diagram of a motion capture system and a compensation device according to an embodiment of the present invention;
2 is a flowchart of a motion capture correction method according to an embodiment of the present invention;
3 is a detailed flowchart of a motion capture correction method according to an embodiment of the present invention;
4 is a 3D visualization diagram of an ankle sensor in which drift does not occur;
5 is a 3D visualization diagram of an ankle sensor in which a drift has occurred;
6 is a 3D view of a standing posture with an ankle twisted;
7 is a diagram showing a distance error when walking with an ankle twisted;
8 is a diagram showing a TPose correction posture in a motion capture system;
9 is a view showing detection by a foot pressure sensor;
10 is a diagram showing the classification of ground contact areas;
11 is a table showing data corresponding to keys and values for ground contact areas;
12 is a diagram showing an extraction method,
13 is a table showing a method of extracting error data;
Fig. 14 is a 3D diagram illustrating the correction of drift by error data.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated and described in the drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms including ordinal numbers such as second and first may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a second element may be termed a first element, and similarly, a first element may be termed a second element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Also, some embodiments may be presented as functional block structures and various processing steps. Some or all of these functional blocks may be implemented as a varying number of hardware and/or software components that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more processors, controllers, or microprocessors, or circuit configurations to perform intended functions. Also, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. Functional blocks may be implemented as an algorithm running on one or more processors. In addition, the present disclosure may employ prior art for electronic environment setting, signal processing, and/or data processing. Terms such as module and component may be used broadly and are not limited to mechanical and physical components.

Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings, but the same or corresponding components regardless of reference numerals are given the same reference numerals, and overlapping descriptions thereof will be omitted.

1 is a block diagram of a motion capture system and correction device according to an embodiment of the present invention, FIG. 2 is a flow chart of a motion capture correction method according to an embodiment of the present invention, and FIG. 3 is a motion capture and compensation method according to an embodiment of the present invention. 4 is a 3D visualization diagram of an ankle sensor in which drift does not occur, FIG. 5 is a 3D visualization diagram of an ankle sensor in which drift occurs, and FIG. 6 is a standing posture with an ankle twisted It is a 3D drawing for , Figure 7 is a view showing a distance error when walking with the ankle twisted, Figure 8 is a view showing the TPose correction posture in the motion capture system, Figure 9 is a view showing the detection by the foot pressure sensor 10 is a diagram showing classification of ground contact parts, FIG. 11 is a table showing data corresponding to keys and values for ground contact parts, and FIG. 12 is a diagram showing an extraction method 13 is a table showing a method of extracting error data, and FIG. 14 is a 3D diagram illustrating drift correction by error data.

Referring to FIG. 1 , a motion capture system 1000 according to an embodiment may include a motion capture device 100 and a sensor 200 .

The motion capture system 1000 may output motion capture through the motion capture device 100 in response to the motion of a subject (eg, a person) equipped with the sensor 200 or process motion information of the subject. Motion capture can be depicted as a 3D drawing.

The sensor 200 may include a gyro sensor, a foot pressure sensor, and the like. Accordingly, the motion capture device 100 may receive motion data, pressure data, joint data (angle), and the like from the sensor 200 .

The motion capture device 100 may be connected to the sensor 200 to process motion data and pressure data. That is, the motion capture device 100 may process and extract motion information by processing data received from the sensor 200 and output a 3D drawing.

As described above, the motion capture device 100 may correct motion capture in addition to processing and extracting motion information. Accordingly, the motion capture device 100 will be described as including a motion capture correction device. Hereinafter, motion capture correction will be described as a standard.

The motion capture compensation device 100 may include a receiver 110 and a controller 120 . The receiver 110 may be connected to the sensor 200 to receive sensor data including motion data and pressure data.

The control unit 120 is connected to the receiver 110, determines the ground contact area using the pressure data received from the foot pressure sensor, and sets a key and value for the ground contact area as a standard of the previously stored data. It is compared with a key and a reference value, and according to the comparison result, the key and value for the ground contact portion may be stored in the pre-stored data or error data may be extracted. . A detailed description of this may be equally applied to a description of a motion capture correction method to be described later. That is, the controller 120 may perform each step of a motion capture correction method described later.

Referring to FIGS. 2 and 3 , the motion capture correction method according to the embodiment includes determining a ground contact area using pressure data received from a foot pressure sensor (S310), a key and value for the ground contact area Comparing (value) with the reference key and reference value of the pre-stored data (S320), and the key and value for the ground contact part according to the comparison result to the pre-stored data It may include storing in or extracting error data (S330).

Furthermore, the step of determining (S310) may include a step of determining whether the ground contact part is both feet or one foot (S311). And the step of extracting (S330) may include a step of determining (S331) whether the key for the ground contact portion and the reference key of the previously stored data are the same.

In addition, in the step of extracting (S330), if the key for the ground contact part and the reference key of the previously stored data are the same, the value of the ground contact part and the previously stored data Determining the error data to which the reference value is reflected (S332) and reflecting the extracted error data to the angle of the ankle (S333) may be included.

In addition, in the step of extracting (S330), if the key for the ground contact part and the reference key of the previously stored data are different, the key and value for the ground contact part It may include adding to the reference key and reference value of the pre-stored data (S334).

Specifically, in the step of determining (S310), the ground contact part may include any one of both feet, one foot, and the air.

Referring to FIG. 10 together, after the motion measurement or motion data reception starts, it is possible to determine a part in contact with the ground (or a foot in contact with the ground) (S310). As an embodiment, the state of the foot using the value applied to the foot pressure sensor (pressure data) may be divided into three states as shown. The three states may include a state with both feet touching the ground (A), a state with one foot touching the ground (B), and a state with both feet off the ground (air, C). For example, the state of C may be a state in which the character (or user) floats in the air (jump, etc.). In the state of C, significant foot pressure may not be measured or received. Accordingly, the comparison step (S320) can be performed when the ground contact part is both feet or one foot. In contrast, in the case of the floating state (C) in the air, the ground contact portion can be judged again. This judgment may be performed at predetermined time intervals.

In the motion capture correction method and apparatus according to the embodiment, a key for the ground contact portion may include a joint angle received from motion data and a pressure value of pressure data for the ground contact portion.

Referring to FIG. 11 together, the motion data may include joint data and joint angles (x, y, z). Alternatively, the joint data may be data extracted based on motion data. And the pressure data may include a pressure value corresponding to the sensor. The value of the ground contact part may include the angle of the ankle in contact with the ground. That is, the value may include ankle data (ankle angle) of the ground contact portion (foot). Also, in the embodiment, sensors are disposed at each position (0 to 14) of the user as shown in FIG. 8, and joint data and joint angles may be derived based on data received from sensors disposed at each position.

Also, in an embodiment, the key and value for the ground contact portion may correspond to received or measured data, eg, current data. For example, when one or more feet (ground contact parts) come into contact with the ground, a key and value as shown in FIG. 11 may be created and stored in some cases.

In particular, in the case of an ankle, since an error of a sensor (eg, a gyro sensor) increases with time, accuracy of an angle of the ankle (or accuracy of joint data) decreases.

Accordingly, the angle of the joint received from the motion data in the comparing step (S320) may be the angle of the joint excluding the ankle.

Afterwards, the step of extracting (S330) may further include a step of determining whether a key for a ground contact portion and a reference key of previously stored data are the same (S331).

Referring to FIG. 12 , it is possible to compare whether a standard key of previously stored data matches a key for a ground contact area. The reference key and value may be formed of the same type of data as the above-described key and value. Furthermore, in the pre-stored data, a reference key and a reference value may correspond to data for a specific motion. In other words, a key for a specific motion may be compared with a key for a currently measured ground contact area.

In determining whether the standard key of the pre-stored data and the measured key are the same or similar, the criterion for determining the same (similar) is the distance of the angle of the motion sensor of the data to be stored, and foot pressure according to the nature of the program. It can depend on how much you specify the sensor pressure interval. That is, the determination of whether they are identical (similar) may change in response to the distance between motion sensor angles and the foot pressure sensor pressure distance.

Furthermore, if the key for the ground contact part and the reference key of the pre-stored data are the same, error data in which the value for the ground contact part and the reference value of the pre-stored data are reflected can be determined. It can (S332).

In an embodiment, referring to FIG. 13 , the error data corresponds to the reference value of the previously stored data corresponding to the reference key of the pre-stored data and the ground contact area corresponding to the key to the ground contact area. It may be a difference in value for

That is, the error data (error angle) may correspond to a value obtained by subtracting a value corresponding to a currently measured or ground contact portion from a value corresponding to a reference key. That is, the error data (or error angle) may correspond to an angle. Accordingly, since the motion data and foot pressure state excluding the ankle already exist (pre-stored data), when error data occurs, it can be easily determined as a situation in which drift has occurred in the ankle.

Thereafter, the control unit may reflect the extracted error data to the angle of the ankle (S333). That is, the error data or the error angle may be continuously reflected in the angle of the ankle. With this configuration, the error angle or error data due to the drift may be applied to or added to or subtracted from the ankle angle measured later. This is updated at predetermined time intervals to make it possible to estimate the ankle state in the initial state (state where no drift has occurred). In other words, the problem of accuracy of values for the same currently measured ground contact area can be solved.

Referring to FIG. 14 , the angle of the ankle on the 3D drawing may change as a drift occurs due to a malfunction of a sensor in a previously stored motion (corresponding to an initial state). Therefore, by applying the error data described above, the previously stored motion and the motion with the error angle added or subtracted may correspond to each other.

Furthermore, if the second error data is calculated after the first error data according to the embodiment is reflected, the second error data instead of the first error data may be reflected. In addition, when the first error data and the second error data are out of a predetermined range, the third error data for the ground contact portion is extracted within a predetermined time after the determination of the second error data, and the difference with the third error data is Small error data can be reflected.

Furthermore, if the key for the ground contact part and the reference key of the pre-stored data are different, the key and value for the ground contact part are set to the reference key and the pre-stored data It can be added to the reference value (S334).

In the embodiment, if the key for the ground contact part and the reference key of the previously stored data are different, the current data (the key for the ground contact part is an operation that has never been reproduced in the past, so the corresponding A key and a value corresponding to the key may be stored, and the motion capture device may repeatedly perform the above-described correction method or correction operation during a period of extracting motion capture.

Additionally, looking at FIGS. 4 and 5 together, in the case of a gyro sensor, since a large error occurs over time, the accuracy of sensor data decreases over time even though it is the same motion, and can be changed from FIG. 4 to FIG. 5 .

Accordingly, the motion capture device and method according to the embodiment of the present invention maps angles of gyro sensors excluding the ankle and pressure data of foot pressure sensors to keys, and data (ankle angle) received from the ankle sensor to values. In this way, the axis of the ankle angle is converted so that values having the same key are determined as previous values in time. That is, if the present method is limited to a situation in which the sole of the foot on the ankle side touches the ground, constant correction for the ankle can be made possible.

6, when standing with the ankle twisted, the ankle goes at an angle that cannot be reproduced by the human body, which can be very unnaturally visualized in 3D. Furthermore, looking at FIG. 7 together, when movement (walking, running, etc.) occurs with the ankle twisted (corresponding to the drift state), the information or data part involved in the movement of the ankle has an error, and the movement direction or It can be seen that there is an error in measuring the distance.

Accordingly, as shown in FIG. 8, a process (Tpose) that is taken to match the value entered into the gyro sensor with the character in 3D space can be performed, but this uses the data received in a specific posture as a correction coefficient and converts the subsequent human joint data to the character. However, in the embodiment, reflection of error data for an angle may be performed during an operation without performing the process (Tpose). That is, the accuracy of 3D visualization can be improved.

The motion capture correction method according to the disclosed embodiment may be implemented by a hardware-implemented processor and a controller.

In addition, the motion capture correction method according to the disclosed embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. In addition, an embodiment of the present disclosure may be a computer-readable recording medium in which one or more programs including instructions for executing a motion capture correction method are recorded.

And, the computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

Here, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium. and temporary storage are not distinguished. For example, a 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, the motion capture correction method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored on a device-readable storage medium such as a memory of a manufacturer's server, an application store server, or a relay server. It can be temporarily stored or created temporarily.

Specifically, it may be implemented as a computer program product including a recording medium in which a program for performing the motion capture correction method according to the disclosed embodiment is stored.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also within the scope of the present invention. belongs to

The term '~unit' used in this embodiment means software or a hardware component such as a field-programmable gate array (FPGA) or ASIC, and '~unit' performs certain roles. However, '~ part' is not limited to software or hardware. '~bu' may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

Although the above has been described with reference to the embodiments, this is only an example and does not limit the present invention, and those skilled in the art to which the present invention belongs will not deviate from the essential characteristics of the present embodiment. It will be appreciated that various variations and applications are possible. For example, each component specifically shown in the embodiment can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present invention as defined in the appended claims.

Claims (11)

Determining a ground contact area using pressure data received from a sensor;
comparing a key and a value of the ground contact portion with a reference key and value of previously stored data; and
and storing a key and a value for the ground contact portion in the previously stored data or extracting error data according to a comparison result.
According to claim 1,
The motion capture correction method of claim 1 , wherein the key for the ground contact portion includes a joint angle received from motion data and a pressure value of pressure data for the ground contact portion.
According to claim 2,
The angle of the joint received from the motion data is an angle of the joint excluding the ankle.
According to claim 3,
The motion capture correction method of claim 1 , wherein the value for the ground contact portion includes an angle of the ankle in contact with the ground.
According to claim 4,
The extraction step is
and determining whether a key for the ground contact portion and a reference key of previously stored data are the same.
According to claim 5,
If the key for the ground contact portion and the reference key of the pre-stored data are the same, the error in which the value for the ground contact portion and the reference value of the pre-stored data are reflected A motion capture correction method comprising: determining data;
According to claim 6,
The error data is
The difference between the reference value of the previously stored data corresponding to the reference key of the pre-stored data and the value for the ground contact portion corresponding to the key for the ground contact portion Motion capture correction method.
According to claim 7,
A motion capture correction method comprising: reflecting the extracted error data to an angle of the ankle.
According to claim 5,
If the key for the ground contact part and the reference key of the pre-stored data are different, the key and value for the ground contact part are set to the reference key of the pre-stored data ) and adding to a reference value.
According to claim 1,
In the determining step, the ground contact area includes any one of both feet, one foot, and the air,
The comparing step is performed when the ground contact part in the determining step is both feet or one foot.
a receiving unit receiving pressure data and motion data from the sensor; and
Including; control unit;
The controller determines the ground contact area using the pressure data received from the sensor, and converts the key and value of the ground contact area to the standard key and value of the previously stored data. A motion capture correction device for comparing, and storing a key and value for the ground contact portion in the previously stored data or extracting error data according to the comparison result.

Images