KR101643690B1 - Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device - Google Patents

Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device Download PDF

Info

Publication number
KR101643690B1
KR101643690B1 KR1020150055689A KR20150055689A KR101643690B1 KR 101643690 B1 KR101643690 B1 KR 101643690B1 KR 1020150055689 A KR1020150055689 A KR 1020150055689A KR 20150055689 A KR20150055689 A KR 20150055689A KR 101643690 B1 KR101643690 B1 KR 101643690B1
Authority
KR
South Korea
Prior art keywords
model
sensor data
portable terminal
gplvm
hmm
Prior art date
Application number
KR1020150055689A
Other languages
Korean (ko)
Inventor
노준용
엄해광
최병국
Original Assignee
한국과학기술원
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 한국과학기술원 filed Critical 한국과학기술원
Priority to KR1020150055689A priority Critical patent/KR101643690B1/en
Application granted granted Critical
Publication of KR101643690B1 publication Critical patent/KR101643690B1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

The present invention provides an apparatus and a method for restoring a walking pose based on a portable terminal, capable of capturing a pose of a portable terminal user by using a motion sensor basically embedded in a portable terminal such as a smartphone, so as to remarkably increase accessibility to motion capture operation. The apparatus comprises: a model learning unit constructing motion models including a hidden Markov model (HMM), a Gaussian process latent variable model (GPLVM), and a multi-layer perceptron (MLP) model, and using training data formed by pose data and sensor data to train the HMM, the GPLVM and the MLP model; a motion sensor embedded in a portable terminal to sense a walking pose of a user holding the portable terminal in three dimension (3D), so as to acquire and output the sensor data; a phase division unit discretizing the sensor data through the HMM and dividing a phase; a potential coordinate point acquisition unit acquiring a potential coordinate point of the phase divided sensor data through the HLP model; and a pose prediction unit predicting the walking pose of the user from the potential coordinate point through the GPLVM.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for restoring a walking posture based on a portable terminal,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion capture system, and more particularly, to a portable terminal-based gait and posture restoration apparatus and method for restoring a walking posture of a user more easily and easily through a mobile terminal.

A motion capture system is used to capture the movement of an actual object and map it to a computer-generated object as a way of animating it. Such a system is essential for the production of video and video games for creating a digital representation of a person used as source data for generating computer graphic (CG) animation.

In a typical system, actors wear costumes with markers attached to different locations (e.g., small reflective markers attached to the torso and limbs), while digital cameras illuminate the markers and move the actors from different angles Record. The system then analyzes the image to determine the location (e.g., spatial coordinates) and orientation of the marker in the actor's garment within each frame. The system tracks the position of the marker to create a spatial representation of the marker over time and builds a digital representation of the moving actor. Such a posture is applied to a digital model, textured and rendered to produce a complete CG representation of the actor and / or act.

However, such a commercial attitude capturing system is a system mainly used in a special effects company that reproduces live-action animation, and thus has a disadvantage in that it requires a complicated and expensive special equipment. In addition, various requirements such as attaching retro-reflective markers to a human body or an additional infrared camera have a problem of imposing space restrictions. That is, the conventional motion capture system has a disadvantage that it is difficult for a general person to use it universally due to cost and spatial constraints.

Accordingly, there is a growing demand for a motion capture system that solves the problem that the user and application fields of the motion capture system are very limited, and allows the non-movie makers and animation producers to easily access and utilize them.

Korean Patent Laid-Open No. 10-2008-0060228

In order to solve the above problems, the present invention can capture the attitude of a user of a portable terminal by using a motion sensor built in a portable terminal such as a smart phone, so that the accessibility to the motion capture operation And to provide a motion capture system and method based on a mobile terminal that can be dramatically increased.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a motion model including a HMM (Hidden Markov Model) model, a GPLVM (Gaussian Process Latent Variable Model) model and an MLP (Multi-Layer Perceptron) A model learning unit that learns the HMM model, the GPLVM model, and the MLP model using training data composed of posture data and sensor data after building; A motion sensor built in a portable terminal for three-dimensionally sensing a walking posture of a user carrying the portable terminal to acquire and output sensor data; A phase divider for dividing the phase of the sensor data after discretizing the sensor data through the HMM model; A potential coordinate point obtaining unit for obtaining potential coordinate points of the phase-separated sensor data through the MLP model; And a posture predicting unit for predicting a user's walking posture from the potential coordinate point through the GPLVM model.

The user walking posture restoration device obtains a feature vector by sampling the new sensor data according to a predetermined time interval, linearly interpolates the sensor data for a portion where data is not sampled, And a sensor data preprocessing unit.

The model learning unit optimizes the nonlinear regression function while inputting the training data N (N is a natural number) under the semantic time condition to the HMM model having a nonlinear regression function supporting mapping between sensor data and latent variables .

The model learning unit generates a matrix C, which is a discrete version of the sensor data S, by calculating the slope of the sensor data for each dimension in every time unit and encodes the slope as an integer value, And optimizing the regression function.

The model learning unit includes:

Figure 112015038499700-pat00001
(N is a natural number) while inputting the training data into the GPLVM model having the mathematical expression expressed by "N ", the potential variable X being the N potential set of variables, Y being the walking posture set Y, the scale matrix W,
Figure 112015038499700-pat00002
,
Figure 112015038499700-pat00003
Is optimized.

The model learning unit includes:

Figure 112015038499700-pat00004
(N is a natural number) while inputting the training data into the MLP model having the mathematical expression expressed by "

According to another aspect of the present invention, there is provided a method of generating a motion model including a HMM (Hidden Markov Model) model, a GPLVM (Gaussian Process Latent Variable Model) model and an MLP (Multi-Layer Perceptron) Building; Learning each of the HMM model, the GPLVM model, and the MLP model using training data composed of attitude data and sensor data; And the sensor data obtained by sensing the user's walking posture three-dimensionally from the motion sensor built in the portable terminal is obtained, the sensor data is discretized through the HMM model, the phase is divided, Obtaining potential coordinate points of the phase-separated sensor data, and predicting a user's walking posture from the potential coordinate points through the GPLVM model.

The portable terminal-based gait restoration apparatus and method according to the present invention can restore the user gait attitude by using only the motion sensor built in the portable terminal, thereby remarkably reducing the cost and various restrictions on the user gait attitude .

FIG. 1 is a view for explaining a walking-posture restoration apparatus based on a portable terminal according to an embodiment of the present invention.
2 is a diagram showing the installation position of the portable terminal and the local coordinate axes of the motion sensor.
3 is a diagram for explaining the operation of the data preprocessing unit according to an embodiment of the present invention.
4 is a diagram for explaining a phase division process according to an embodiment of the present invention.
5 and 6 are views for explaining a data discretization process according to an embodiment of the present invention.
FIGS. 7 and 8 are views for explaining a method for restoring a walking posture based on a portable terminal according to an embodiment of the present invention.
FIG. 9 and FIG. 10 are views for explaining the effect of the method of restoring the walking posture based on the portable terminal according to the embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

The following terms are defined in consideration of the functions of the present invention, and these may be changed according to the intention of the user, the operator, or the like.

The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art to which the present invention pertains. Only. Therefore, the definition should be based on the contents throughout this specification.

Before describing the present invention, the operation principle of the present invention will be briefly described in order to facilitate understanding of the present invention.

The user's walking / restoring device of the present invention is widely used and is a low-cost, portable device capable of being portable anytime and anywhere (for example, smart phone, tablet PC, smart watch, etc.) And does not require high-level sensor data.

However, in order to express the user's walking posture using one smartphone, the present invention learns a motion model based on high-dimensional posture data obtained in advance, and then, based on the learned motion model, So that new sensor data can be recorded by any person at anytime and anywhere without any space constraint.

In the present invention, in order to solve some problems that may occur when direct mapping between the high dimensional attitude data and the low dimensional sensor data is performed, a HMM (Hidden Markov Model) model, a GPLVM (Gaussian Process Latent Variable Model) -layer perceptron) model to construct a motion model.

The low-dimensional motion model according to the present invention enables to grasp the user's walking posture in response to new low-dimensional sensor data. Low-level embedding is performed via GPLVM, and GPLVM generates a new potential space in which each posture is effectively spread based on the correlation between each posture. The latent coordinates defined by the nonlinear probability model such as GPLVM So that robust mapping to noise can be performed.

The restoration of the walking posture according to the sensor data may cause a problem of ambiguity and temporal inconsistency. Therefore, in order to prevent the occurrence of such a problem, the present invention performs phase division on new sensor data provided from the portable terminal through the HMM model . According to the learned HMM model, the phase of each frame can be automatically identified using a moving window window, and the MLP model is applied to define the mapping between the potential points belonging to the phase and the sensor data for each phase.

Accordingly, the present invention can grasp the human pedestrian attitude through one portable terminal and perform the pedestrian attitude restoration operation without spatial constraint or special equipment, but it is possible to guarantee a high quality.

Also, since the statistical data model generates a reasonable interpolation result for the missing data, it does not require a large amount of data. In addition, the mapping algorithm of the present invention can effectively associate pre-recorded attitude clips with free sensor data with limited degrees of freedom based on low-dimensional embedding.

Further, the present invention constructs a state decomposition model to activate learning of the semantic level, thereby activating the phase classification, thereby handling posture ambiguity such as shaking.

FIG. 1 is a view for explaining a walking-posture restoration apparatus based on a portable terminal according to an embodiment of the present invention.

Referring to FIG. 1, the user walking / restoring apparatus of the present invention includes a training data storage unit 10 for storing a plurality of training data composed of high dimensional attitude data and low dimensional sensor data, A HMM (Hidden Markov Model) model 31, a GPLVM (Gaussian Process Latent Variable Model) model 32 and an MLP (Multi-Layer Perceptron) model 33 for sensing and outputting sensor data ), A model learning unit (30) for constructing a motion model composed of a plurality of training data and learning a motion model through the plurality of training data, a walking posture restoring unit for restoring a walking posture from new sensor data acquired through the motion sensor (40) and the like.

When the new sensor data is acquired, the gait restoring unit 40 obtains the feature vector by sampling the new sensor data according to a predetermined time interval, linearly interpolates the sensor data with respect to the data not sampled, A phase divider 42 for dividing the phase of the new sensor data through the HMM model 31 and a phase divider 42 for dividing the phase of the new sensor data through the HMM model 31, A potential coordinate point obtaining unit 43 for obtaining potential coordinate points of the divided sensor data, a pedestrian-posture predicting unit 44 for predicting a user walking posture from the potential coordinate points through the GPLVM model 33, and the like Lt; / RTI >

In addition, if necessary, an animation conversion unit for performing a general animation conversion operation including inverse kinematics and Footskate cleanup for automatically correcting the synchronized animation to smoothly satisfy additional constraints such as required foot positions, (45), and the like.

The training data composed of the high dimensional attitude data and the low dimensional sensor data can be acquired and stored for each of various postures such as straight walk, jumping, and the like. Preferably, the attitude data may be acquired and stored via existing commercial motion capture systems (e.g., a beacon system comprised of eight 120 Hz MX-40 cameras), and the sensor data may be acquired by a motion sensor As shown in FIG. It is desirable that several hundreds of these are acquired and stored for each walking posture.

The motion sensor 20 of the present invention can be implemented by an accelerometer and a gyroscope built in various portable terminals. M / s 2 when the sensor data is acquired through the accelerometer, and rad / s data unit when acquired through the gyroscope, all of which are recorded corresponding to the local coordinate system of the motion sensor.

FIG. 2 is a view showing the installation position of the portable terminal and the local coordinate axes of the motion sensor. Referring to FIG. 2, the user attaches a portable terminal to the lower end of his leg to acquire sensor data corresponding to his / her walking motion. The sensor data has an XYZ axis in the case of an accelerometer and a roll, yaw and pitch axis in the case of a gyroscope. The corresponding feature vector can reflect the user's posture .

Hereinafter, a learning method of the motion model of the present invention will be described with reference to FIGS. 2 to 6. FIG.

As described above, the present invention trains a motion model using pre-stored high dimensional three-dimensional attitude data and low-dimensional sensor data synchronized therewith.

The motion model is constructed using the HMM model 31, the MLP model 32, and the GPLVM model 33, and the GPLVM model 33 considering the dimension of the attitude data is relatively larger than the dimension of the sensor data. Learning is performed so that the walking posture data can be represented and restored with low dimensional data, and the potential points and sensor data resulting from the GPLVM model 33 are learned using the MLP model 32. [ In order to calibrate the smoothness of the flow and posture, we study the HMM model which is an algorithm of time series analysis by walking position.

The user's walking posture generally requires a high-level description of the character posture, and it is difficult to directly match the low-dimensional sensor data obtained through the motion sensor of the portable terminal. In the present invention, a new low-dimensional space representing high-dimensional training attitude examples is created. Such low-dimensional space is known as LEVINE S., WANG J. M., HARAUX A., POPOVI ㅄ C Z., KOLTUN V .: Continuous character control with low-dimensional embeddings. Dimensional embedding technique described in ACM Transactions on Graphics (TOG) 31, 4 (2012), 28. 2, 3,

The GPLVM model 13 represents attitude data having latent coordinate points, and the nonlinear regression method can relate these potential variables to the sensor data obtained through the portable terminal.

- Learning GPLVM (Gaussian Process Latent Variable Model) Model

The posture vector y is composed of the channel values of the joint angles including the root. In the present invention, in order to define the mapping from the low dimensional latent variable x to the posture vector y, the Gaussian process ) Model. After training the GP model through the data (x, y) corresponding to each other, a new vector y corresponding to the new input vector x is calculated through the GP model.

In order to process different dispersion ranges for each joint, the channels of the attitude vector are divided into a diagonal scale matrix

Figure 112015038499700-pat00005
(Where d y is the dimension of Y).

In order to grasp the correlation between data,

Figure 112015038499700-pat00006
(RBF) kernel of Equation (1) is selected.

[Equation 1]

Figure 112015038499700-pat00007

At this time,

Figure 112015038499700-pat00008
Is 1 if i and j are equal, and 0 otherwise.

According to the RBF kernel, as the distance between two potential points increases, the correlation decreases gradually.

Kernel Matrix

Figure 112015038499700-pat00009
The kernel matrix
Figure 112015038499700-pat00010
.

The log likelihood term for the posture is

Figure 112015038499700-pat00011
, Which is proportional to the following equation.

&Quot; (2) "

Figure 112015038499700-pat00012

Here, N is the number of training data, and Y = [y 1 , ... , y N ] T , X = [x 1 , ... , x N ] T.

A time frame mismatch may occur if the frame is calculated by the frame.

Including the root position is also important for restoring the complete walking posture, but including the relative skeleton transformation directly to y can result in additional errors such as sliding effects or inaccurate mapping results.

In order to solve this problem, the degree of motion per time interval is used in the present invention. Speed matrix

Figure 112015038499700-pat00013
To calculate a new GP in the velocity domain. Here, the skeletal velocity in the form of a horizontal spatial transformation is also considered.

The log likelihood of the speed GP requires a new kernel function as shown in Equation 3 below.

&Quot; (3) "

Figure 112015038499700-pat00014

At this time,

Figure 112015038499700-pat00015
Wow
Figure 112015038499700-pat00016
Are parameters to be optimized.

Similar to equation (2), the likelihood ratio value is proportional to (4) and (5).

&Quot; (4) "

Figure 112015038499700-pat00017

&Quot; (5) "

Figure 112015038499700-pat00018

At this time,

Figure 112015038499700-pat00019
And
Figure 112015038499700-pat00020
Is a scale matrix.

In this way, introducing the speed GP model in the learning process results in a better embedding result and more reasonable character motion.

The setting of the learning process depends on the latent variable X, the kernel function

Figure 112015038499700-pat00021
,
Figure 112015038499700-pat00022
Hyperparameters, scale matrix < RTI ID = 0.0 >
Figure 112015038499700-pat00023
,
Figure 112015038499700-pat00024
. The input attitude data (Y) and its velocity
Figure 112015038499700-pat00025
Maximizing the log posterior is achieved by Equation (6).

&Quot; (6) "

Figure 112015038499700-pat00026

At this time,

Figure 112015038499700-pat00027
And
Figure 112015038499700-pat00028
Is a posture and velocity likelihood ratio calculated from Equations (2) and (4), and represents original posture information.

Figure 112015038499700-pat00029
Wow
Figure 112015038499700-pat00030
Figure 112015038499700-pat00031
Is the prior terms for the parameters from the kernel function. The logarithm is maximized by applying a Limited-memory Broyden algorithm (LBFGS), with each degree of likelihood and priority for parameters calculated at this stage.

As a result, the GPLVM model of the present invention can be finally expressed as in Equation (7).

&Quot; (7) "

Figure 112015038499700-pat00032

- Learning HMM model

The sensor data S with sensor data S, which is the N ㅧ K matrix of k sensor data dimensions and the potential variable X associated therewith, assumes that the mapping between S and X using standard nonlinear regression techniques is data independent and equally distributed The result that can not be done is derived.

In order to accurately describe the sequential nature of the data, nonlinear regression functions that support mapping between sensor data S and latent variable X must be trained under semantic time conditions to ensure temporal consistency.

In the present invention, as shown in FIG. 4, the training sequence can be divided into several steps based on the ground contact state of the foot. Each color represents a different phase from the two-dimensional potential space derived from the GPLVM model learned from the two-step working motion clip of FIG. In other words, the orange starts to walk until the left foot is removed from the ground, the green moves from the left foot (when walking) until the right arm is removed from the ground, the turquoise moves from the right foot (while walking) until the left foot is removed, , And gray represents the phase corresponding to each of the left foot movements to stop walking. This phase division allows phase information to be added in the posture reconstruction step, thereby preventing the possibility of ambiguity in advance.

In order to more easily train the HMM model, it is first discretized using predetermined features instead of the actual value of the sensor data S. In general, if the input to the HMM is continuous, the input is discretized and the discretized input is used as a training feature. In the present invention, in order to construct the discrete version of the sensor data S, matrix C, The slope of the sensor data for each dimension is calculated at every time unit, and the slope is encoded as an integer value to generate a matrix C, which is a discrete version of the sensor data S.

The discretization scheme of the present invention takes into account the temporal tendency of each phase so that it can sufficiently express the distinctive feature vector over a small number of feature values (e.g., three) and use it to compare a particular phase with another phase It can help.

The HMM must be trained for each phase, and for each training, C is relocated to C 'based on the class label of the manually identified phase. As shown in FIG. 6, C 'is finally composed of an mc i block matrix, where m is the number of phases and each row of c i is a discretized data vector classified into the i th phase. At this time, the c i block matrix includes semantically the same type of postures, and by applying the HMM to each c i block matrix, it is possible to acquire m number of HMM classifiers used for evaluating new sensor data.

- Learning MLP Model

The mapping between the sensor data of the portable terminal and the potential coordinate point can be defined by optimizing the low dimensional potential variable X representing the whole body posture Y and training the MLP model based on the low dimensional potential variable X.

Even in low dimensional space, linear mapping can generally lead to bad results. Therefore, in the present invention,

Figure 112015038499700-pat00033
To define the mapping between sensor data and potential coordinate points, where p i is the MLP model trained by X for each element of C and C.

The sensor data preprocessing unit 31 samples the sensor data at a predetermined time interval (for example, 30 fps) to acquire the feature vectors, and linearly interpolates the sensor data for the data not sampled. The sampled data is removed through a low-pass filter or the like, and then supplied to the model learning unit 30 together with attitude data. At this time, it is preferable that the low-pass filter is a one-dimensional Gaussian filter. By performing one-dimensional Gaussian filtering for each dimension, the noise included in the sensor data can be effectively removed as shown in FIG.

Hereinafter, the walking posture restoration method of the present invention will be described with reference to FIGS. 7 and 8. FIG.

First, in step S1, when new sensor data is acquired through the motion sensor of the portable terminal, sampling is performed according to a predetermined time interval to acquire a feature vector, and the sensor data for a part where data is not sampled is linearly interpolated , And S new obtained by low-pass filtering is obtained and stored. By this data preprocessing process, new sensor data have robust characteristics against various noise.

Then, in step S2, S new is discretized through the HMM model 31 and then phase-divided to generate C new .

The HMM model 31 includes a Z-size window

Figure 112015038499700-pat00034
To obtain phase information for all frame values. As shown in Fig. 8, the window U is moved in units of frames, and for each dimension of the sensor data
Figure 112015038499700-pat00035
And determines the phase having the highest probability value as the phase of the window U.

In step S3, C new is plugged into P mlp through the MLP model and converted to potential coordinate point X new .

In step S4, the GPLVM model (

Figure 112015038499700-pat00036
) To predict a new walking posture Y new relative to the potential coordinate point X new .

&Quot; (7) "

Figure 112015038499700-pat00037

In this case, k (x) is an N * 1 vector composed of the i-th element of k rbf (x new , x i ), and x new and y new are components of X new and Y new in each frame. The bias condition is a d y * 1 vector determined by the row direction average of all Y vectors used in the learning.

However, since the corresponding operation is performed independently for each frame, noise may be induced over time. In the present invention, the Laplacian smoothing is applied to ensure the temporal consistency of angular values using a SLERP algorithm in a quaternion space. In addition, to prevent foot skating artifacts from occurring, the skeletal transition to foot contact is adjusted, and foot contact is automatically detected when the foot is close enough to the ground and its velocity is below a threshold.

Hereinafter, the effects of the present invention will be described with reference to FIGS. 9 to 10. FIG.

At this time, the motion model is trained using the two-step walking motion clip of 136 frames, the HMM is trained with the clip having 487 frames expressed in four similar postures, and the potential for generation of the motion model I set the dimension of the coordinates to five. In the HMM, the size of the window is set to 10 - 20 in consideration of the motion speed, and the level of discretization is set to 3.

FIG. 9 shows a comparison of the walking posture using the Vicon MX-40 120 Hz 3D motion capture camera with the predicted walking posture through the GPLVM model of the present invention. In FIG. 9, blue indicates an actual walking posture, Indicates posture.

(a) shows the predicted walking posture using only a simple linear regression algorithm. Referring to this, a considerable error occurs between the actual walking posture and the predicted walking posture due to the mapping between the low-dimensional and high dimensional data Able to know.

(b) shows the predicted walking posture using the GPLVM and the linear regression algorithm together, which is better than the result of (a), but the direction of each pose is still unstable.

(c) shows the predicted walking posture using the GPLVM and the MLP, which is a nonlinear regression algorithm, in which the leg motion is represented more realistically but the temporal mismatch occurs and the shaking phenomenon occurs in some frames Able to know.

(d) shows the predicted walking posture using the GPLVM and HMM, which performs phase division in addition to the nonlinear regression algorithm MLP, which shows the final stabilization result produced by integrating the state decomposition.

That is, as shown in FIG. 10, when restoring the walking posture using the GPLVM, the MLP, and the HMM together, it can be seen that the error with the actual walking posture is significantly reduced.

The computer-readable recording medium on which the program commands are recorded may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, Media storage devices.

The computer-readable recording medium on which the above-described program is recorded may be distributed to a computer apparatus connected via a network so that computer-readable codes can be stored and executed in a distributed manner. In this case, one or more of the plurality of distributed computers may execute some of the functions presented above and send the results of the execution to one or more of the other distributed computers, The computer may also perform some of the functions described above and provide the results to other distributed computers as well.

The computer capable of reading the recording medium on which the application as the program for driving the walking and posture restoring apparatus and method based on the portable terminal according to each embodiment of the present invention is read may be not only a general PC such as a general desktop or a notebook computer, A mobile terminal such as a cellular phone, a tablet PC, a PDA (Personal Digital Assistants), and a mobile communication terminal. In addition to this, it should be interpreted as all devices capable of computing.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. The codes and code segments constituting the computer program may be easily deduced by those skilled in the art. Such a computer program can be stored in a computer-readable storage medium, readable and executed by a computer, thereby realizing an embodiment of the present invention. As a storage medium of the computer program, a magnetic recording medium, an optical recording medium, or the like can be included.

It is also to be understood that the terms such as " comprises, "" comprising," or "having ", as used herein, mean that a component can be implanted unless specifically stated to the contrary. But should be construed as including other elements. All terms, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Commonly used terms, such as predefined terms, should be interpreted to be consistent with the contextual meanings of the related art, and are not to be construed as ideal or overly formal, unless expressly defined to the contrary.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (8)

A motion model including a HMM (Hidden Markov Model) model, a GPLVM (Gaussian Process Latent Variable Model) model, and a MLP (Multi-Layer Perceptron) model is constructed and then training data composed of attitude data and sensor data are used to generate the HMM A model learning unit for learning each of the model, the GPLVM model, and the MLP model;
A motion sensor built in a portable terminal for three-dimensionally sensing a walking posture of a user carrying the portable terminal to acquire and output sensor data;
A phase divider for dividing the phase of the sensor data after discretizing the sensor data through the HMM model;
A potential coordinate point obtaining unit for obtaining potential coordinate points of the phase-separated sensor data through the MLP model; And
And a posture predicting unit for predicting a user's walking posture from the potential coordinate point through the GPLVM model. The method according to claim 1,
And a sensor data preprocessing unit for linearly interpolating the sensor data for a portion where the new sensor data is sampled at a predetermined time interval to acquire a feature vector, Wherein said portable terminal is a walking / resting device. The apparatus of claim 1, wherein the model learning unit
Wherein the nonlinear regression function is optimized while inputting N (N is a natural number) training data under the semantic time condition to the HMM model having a nonlinear regression function supporting mapping between sensor data and latent variables. Based gait restoring device. 4. The apparatus of claim 3, wherein the model learning unit
After generating a matrix C, which is a discrete version of the sensor data S, by calculating the slope of the sensor data for each dimension in every time unit and encoding the slope to an integer value, the nonlinear regression function is optimized through the matrix C Wherein said portable terminal-based gait restoring device is configured to detect a gait of said portable terminal. The apparatus of claim 1, wherein the model learning unit
"
Figure 112015038499700-pat00038
(N is a natural number) while inputting the training data into the GPLVM model having the mathematical expression expressed by "N ", the potential variable X being the N potential set of variables, Y being the walking posture set Y, the scale matrix W,
Figure 112015038499700-pat00039
,
Figure 112015038499700-pat00040
Wherein the portable posture restoration device optimizes the walking posture restoration device. The apparatus of claim 1, wherein the model learning unit
"
Figure 112016049758600-pat00041
(N is a natural number) of the training data in the MLP model having the mathematical expression expressed by "n " (where N is a natural number). delete Constructing a motion model including a HMM (Hidden Markov Model) model, a GPLVM (Gaussian Process Latent Variable Model) model, and a MLP (Multi-Layer Perceptron) model;
Learning each of the HMM model, the GPLVM model, and the MLP model using training data composed of attitude data and sensor data; And
When the sensor data obtained by sensing the user's walking posture three-dimensionally from the motion sensor built in the portable terminal is obtained, the sensor data is discretized through the HMM model, the phase is divided, and the phase Obtaining potential coordinate points of the divided sensor data, and predicting a user's walking posture from the potential coordinate points through the GPLVM model.
KR1020150055689A 2015-04-21 2015-04-21 Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device KR101643690B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150055689A KR101643690B1 (en) 2015-04-21 2015-04-21 Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150055689A KR101643690B1 (en) 2015-04-21 2015-04-21 Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device

Publications (1)

Publication Number Publication Date
KR101643690B1 true KR101643690B1 (en) 2016-08-10

Family

ID=56713505

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150055689A KR101643690B1 (en) 2015-04-21 2015-04-21 Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device

Country Status (1)

Country Link
KR (1) KR101643690B1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107292250A (en) * 2017-05-31 2017-10-24 西安科技大学 A kind of gait recognition method based on deep neural network
CN115620397A (en) * 2022-11-07 2023-01-17 江苏北斗星通汽车电子有限公司 Vehicle-mounted gesture recognition system based on Leapmotion sensor

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080060228A (en) 2005-08-26 2008-07-01 소니 가부시끼 가이샤 Motion capture using primary and secondary markers

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080060228A (en) 2005-08-26 2008-07-01 소니 가부시끼 가이샤 Motion capture using primary and secondary markers

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Deena, Salil, and Aphrodite Galata. "Speech-driven facial animation using a shared Gaussian process latent variable model." Advances in Visual Computing. Springer Berlin Heidelberg, 2009. 89-100. *
Kuli263;, Dana, Danica Kragic, and Volker Kruger. "Learning action primitives." Visual analysis of humans. Springer London, 2011. 333-353. *
Kulić, Dana, Danica Kragic, and Volker Kruger. "Learning action primitives." Visual analysis of humans. Springer London, 2011. 333-353.

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107292250A (en) * 2017-05-31 2017-10-24 西安科技大学 A kind of gait recognition method based on deep neural network
CN115620397A (en) * 2022-11-07 2023-01-17 江苏北斗星通汽车电子有限公司 Vehicle-mounted gesture recognition system based on Leapmotion sensor

Similar Documents

Publication Publication Date Title
US11908057B2 (en) Image regularization and retargeting system
Von Marcard et al. Sparse inertial poser: Automatic 3d human pose estimation from sparse imus
CN105981075B (en) Utilize the efficient facial landmark tracking in wire shaped homing method
KR101135186B1 (en) System and method for interactive and real-time augmented reality, and the recording media storing the program performing the said method
JP6367926B2 (en) Augmented reality (AR) capture and play
EP2880633B1 (en) Animating objects using the human body
US11030773B2 (en) Hand tracking based on articulated distance field
US9747668B2 (en) Reconstruction of articulated objects from a moving camera
CN113963445B (en) Pedestrian falling action recognition method and equipment based on gesture estimation
KR20120006610A (en) System and method of face animation
KR102270949B1 (en) 3d human body model reconstruction apparatus and method
KR20130110441A (en) Body gesture recognition method and apparatus
JP2012141881A (en) Human body motion estimation device, human body motion estimation method and computer program
US9870637B2 (en) Frame removal and replacement for stop-action animation
KR101643690B1 (en) Apparatus and method for reconstruction of human locomotion by using motion sensor embedding a portable device
JP2012073852A (en) Human body attitude estimation device, human body attitude estimation method, and computer program
CN112241001B (en) Radar human body action recognition method, radar human body action recognition device, electronic equipment and storage medium
Eom et al. Data‐Driven Reconstruction of Human Locomotion Using a Single Smartphone
KR101515845B1 (en) Method and device for gesture recognition
US20230104702A1 (en) Transformer-based shape models
US20240037826A9 (en) Image Regularization and Retargeting System
Zeng et al. Motion capture and reconstruction based on depth information using Kinect
Han et al. Controlling virtual world by the real world devices with an MPEG-V framework
US11341703B2 (en) Methods and systems for generating an animation control rig
Yang Application of Single Human Pose Estimation: Generating Real-time Character Animation

Legal Events

Date Code Title Description
E701 Decision to grant or registration of patent right
GRNT Written decision to grant
FPAY Annual fee payment

Payment date: 20190625

Year of fee payment: 4