KR20210079542A - User Motion Recognition Method and System using 3D Skeleton Information - Google Patents

Abstract

Provided are a user motion recognition method using 3D skeleton information, and a system thereof. The motion recognition method according to an embodiment of the present invention comprises the steps of: detecting a person in a current frame; extracting joints of the detected person; classifying the extracted joints into a plurality of joint sets; calculating spatial characteristics of the skeleton based on distance information about a distance between the classified joint sets; calculating spatial-temporal characteristics in which temporal characteristics are added to the spatial characteristics of the skeleton based on the calculated spatial characteristics and a change in position between frames of the core joints; and inferring human motion by inputting the calculated spatial-temporal characteristics to an artificial intelligence model trained for human motion recognition. Accordingly, the method extracts the user's 3D joint information from the continuous 2D images, calculates the spatial characteristics of the skeleton, and then adds the temporal characteristics that appear as a change in the position of each joint within a motion section to recognize the user's motion. Therefore, it is possible to recognize/estimate the user's motion only by the dynamic calculation of the 3D skeleton information.

Description

User Motion Recognition Method and System using 3D Skeleton Information

The present invention relates to image processing technology, and more particularly, to a method and system for recognizing/estimating a user's motion using skeletal information.

Motion recognition technology is a technology for estimating the movement of a person in an input image, and is being applied to various industrial fields, such as social robots and interactive interfaces between computers and users.

With the recent development of a 3D distance sensor (eg, Kinect) and deep learning technology, it is possible to estimate a human skeletal structure and a movement pattern from a two-dimensional image, and overcome the difficulties of motion recognition to a large extent.

However, the method using the 3D distance sensor has limitations in using various images on the Internet. In addition, the existing motion estimation method using template matching, histogram, etc. has a disadvantage in that it requires additional elements in addition to human joint information.

The present invention has been devised to solve the above problems, and an object of the present invention is to provide a method and system for recognizing/estimating a user's motion only by mechanical calculation of 3D skeletal information.

According to an embodiment of the present invention for achieving the above object, a motion recognition method includes: detecting a person in a current frame; extracting the detected human joints; classifying the extracted joints into a plurality of joint sets; calculating spatial characteristics of the skeleton based on distance information between the classified joint sets; calculating a spatio-spatial feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint; and inputting the calculated spatio-temporal characteristics to the artificial intelligence model trained for human motion recognition, and inferring the human motion.

The classification step may include rearranging the extracted joints in consideration of the physical connectivity of the extracted joints; It may include; classifying the rearranged joints into a plurality of joint sets.

The extraction step may include extracting 3D joint coordinates, and the classification step may include normalizing the extracted 3D joint coordinates; Calculating the center of gravity of the skeleton from the normalized 3D joint coordinates; further comprising, based on the calculated center of gravity, rearranging the joints in consideration of the physical connectivity of the joints, based on the calculated center of gravity, relocating It may be to classify the acquired joints into a plurality of joint sets.

The step of calculating the spatial characteristics of the skeleton may be to calculate the spatial characteristics by reflecting the intrinsic distance information between the classified joint sets in the physical distance information between the classified joint sets.

The physical distance information may be a relative distance value between joints, and the intrinsic distance information may be determined by a relationship between joint sets including joints.

The spatio-temporal feature calculation step may be to calculate the spatio-temporal feature by reflecting the change in the position of the key joint in the time window including the current frame to the spatial feature.

The spatio-temporal feature calculation step includes: selecting a core joint from among the extracted joints; calculating the inter-frame position change of the core joint; Selecting the core frames that distinguish the motion of a person with reference to the calculated position change; It may further include; setting a time window based on the selected key frames.

The selection step may be to select the end joints of the skeleton as core joints.

The spatio-temporal feature calculation step may be to extract N frames in the order of the largest change in the position of key joints in the time window including the current frame, and combine the spatio-temporal features calculated using the extracted frames into one. have.

According to another aspect of the present invention, a camera for generating an image; Detects a person in the current frame of the image, extracts the detected human joints, classifies the extracted joints into a plurality of joint sets, calculates the spatial characteristics of the skeleton based on distance information between the classified joint sets, and calculates Based on the spatial characteristics and the position change between the frames of the core joints, a spatio-temporal feature with temporal features added to the spatial features of the skeleton is calculated, and the calculated spatio-temporal features are applied to the artificial intelligence learned for human motion recognition. A motion recognition system is provided, comprising: a computing system that infers a human motion by inputting it into a model.

According to another aspect of the present invention, the method comprising: classifying human joints detected in a current frame into a plurality of joint sets; calculating spatial characteristics of the skeleton based on distance information between the classified joint sets; calculating a spatio-spatial feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint; Inferring a human motion by using the calculated spatio-temporal features; a method for recognizing a motion is provided, comprising: a.

According to another aspect of the present invention, the method comprising: classifying human joints detected in a current frame into a plurality of joint sets; calculating spatial characteristics of the skeleton based on distance information between the classified joint sets; calculating a spatio-spatial feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint; A computer-readable recording medium is provided in which a program capable of performing a motion recognition method, comprising: inferring a human motion by using the calculated spatio-temporal characteristics.

As described above, according to the embodiments of the present invention, after extracting the user's 3D joint information from the continuous 2D image and calculating the spatial characteristics of the skeleton, the temporal characteristics appearing as a change in the position of each joint within the motion section are obtained. In addition, by recognizing the user's motion, it is possible to recognize/estimate the user's motion only by the dynamic calculation of 3D skeleton information.

1 is a result of classifying a specific action of a user within each image from images captured by a camera;
2 is a flowchart provided for explaining a user gesture recognition method according to an embodiment of the present invention;
3 is a detailed flowchart of the joint classification process;
4 is a result of rearranging 14 joint nodes,
Figure 5 is a graphical representation of Figure 4,
6 is a joint set classification result,
7 is a detailed flowchart of the process of calculating the spatio-temporal characteristics by reflecting the change in the position of the core joints in the spatial characteristics, and,
8 is a block diagram of a gesture recognition system according to another embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

In an embodiment of the present invention, a method and system for recognizing a user's motion using 3D skeleton information are provided.

In the motion recognition method according to an embodiment of the present invention, the user's 3D joint information is extracted from the continuous 2D image, the spatial characteristics of the skeleton are calculated, and then the temporal characteristics indicated by the position change of each joint within the motion section are added. Recognize the user's actions.

In an embodiment of the present invention, the user's joint information is extracted based on deep learning and motion is recognized, and the user's accurate posture estimation and correction is achieved by mapping the user's joint features to the demonstration image, not by comparing similarity with the demonstration motion. It is possible.

In addition, in the embodiment of the present invention, unlike the existing methodologies such as template matching and histogram, the user's motion is estimated using only joint information, and learning is possible with a small amount of database, and the learning time and processing speed are higher than before. fast.

A method of constructing learning images necessary for user gesture recognition is as follows.

First, as shown in FIG. 1 , a specific action of a user is classified in each image from images captured by one or more cameras. At this time, since most of the captured images have several motions mixed, the user's motion in each image should be classified as a single motion. From the image obtained in this way, it is classified into various categories such as daily motion, exercise, housework, etc. according to the motion sequence of the person, and each category consists of sub-categories.

For example, within the 'daily action' category, it may be re-classified into sub-categories such as 'drinking water' and 'reading a newspaper'. In addition, specific additional motions can continuously expand the categories required for motion recognition by taking their own video.

On the other hand, in order to reflect the positional characteristics of the joint according to the single viewpoint and the multi-viewpoint, an official database (DB) may be used.

Hereinafter, a method of recognizing a user's motion will be described in detail with reference to FIG. 2 . 2 is a flowchart provided to explain a method for recognizing a user's motion according to an embodiment of the present invention.

As shown, first, a person is detected in the current frame (S110), and joints of the detected person are extracted (S120). In step S120, 14 3D joint coordinates are extracted using a deep learning-based joint estimator.

Next, the joints extracted in step S120 are classified into a plurality of joint sets (S130). Step S130 will be described in detail with reference to FIG. 3 . 3 is a detailed flowchart of a joint classification process.

As shown, first, the 3D joint coordinates extracted in step S120 are normalized to a value between -1 and 1 (S131), and the center of gravity of the skeleton is calculated from the normalized 3D joint coordinates (S132).

Because the range value of the joint coordinates in the image is different due to the distance between the camera and the user, it is necessary to normalize the range of the joint coordinates in step S131.

Next, taking the calculated center of gravity as the origin and considering the physical connectivity of the extracted joints, the extracted joints are rearranged (S133), and the relocated joints are classified into a plurality of joint sets based on the calculated center of gravity (S134). ).

Accordingly, it is possible to use the result of the absolute position of the skeletal center rather than the relative position information of the user in the image.

4 shows the results of defining the annotation of the joint coordinates and rearranging 14 joint nodes with the center of gravity of the skeleton as the origin (start point). And, FIG. 5 shows the physical connectivity between the joints shown in FIG. 4 as a graph. In addition, the results of classifying the joints into five joint sets in consideration of the physical connectivity from the center of gravity (start node) of the skeleton shown in FIG. 5 to each end point (end node) are shown in FIG. 6 .

Again, it will be described with reference to FIG. 2 .

After classifying the joints into sets. Based on the classified distance information between the joint sets, the spatial characteristics of the skeleton are calculated (S140). The spatial characteristics of the skeleton are calculated by estimating physical distance information between the classified joint sets, and reflecting intrinsic distance information between the classified joint sets in the estimated physical distance information.

Euclidean distances between all joint coordinates can be calculated and the relative distance values between joints can be estimated from this. At this time, when the missing coordinates are included in the two joint coordinates in the distance calculation, the overall distance weight may be biased, and thus, it is defined as “0” to minimize the influence thereof.

와 같이 표현한다. And, in order to reflect the implicit (implicit) weight between each joint, the intrinsic distance information between the joint nodes is obtained based on the graph hierarchical structure of FIG.

express it as

The joint node Ji is defined as the minimum distance between the ith joint node (Ji) from the center of gravity (J0) of the skeleton, and it is assumed that the weight values of all edges of the graph are “1”. For example, when two specific nodes (i and j joint nodes) are included in the same joint set, distance information between the two nodes is expressed as |Ji-Jj|. In addition, when included in different joint sets, the distance information between two nodes is always expressed as the minimum distance between the node and the center of gravity rather than the distance weight between the two nodes.

Therefore, the intrinsic distance weight between the final two joint nodes is defined as min(Ji, Jj, |Ji-Jj|), and in the embodiment of the present invention, the spatial characteristics of the joint include physical distance information and implicit (implicit) distance information. it will be combined

Next, based on the calculated spatial characteristics and the position change between the frames of the core joint, a spatio-temporal characteristic in which a temporal characteristic is added to the spatial characteristic of the skeleton is calculated (S150).

Step S150 is to extract the core motion section from the entire motion section, and reflect the change in the position of the core joint in the extracted core motion section to the spatial feature to calculate the spatio-temporal feature, and the specific process is shown in FIG. 7 .

For spatio-temporal feature calculation, as shown in FIG. 7 , a core joint is selected among the extracted joints ( S151 ).

Since the actual human motion has continuity, it is difficult to estimate the entire motion from a single image included in the motion section. Therefore, it is necessary to construct a data recognition model in consideration of the correlation between single frames constituting the entire image.

To this end, in the embodiment of the present invention, continuous motion is recognized by converting the positional change between each corresponding joint according to time change into a time weight along with the user's spatial information from two or more input images.

First, the position of the joint changes according to the change in motion, and the correlation between the joint sets is changed accordingly. For example, in the 'walking' motion, the 'wrist' joint and the 'ankle' joint show a high correlation, but the 'wrist' joint and other joints show a low correlation.

Also, the joint sets contributing to determining the 'walking' motion and the 'calling' motion are different from each other. Therefore, in order to determine the change in motion in the image, it is necessary to understand the correlation of the joint set according to the motion type.

Therefore, a relation element between all joints is not required to estimate the user's continuous motion, and the entire joint set does not affect all motion decisions. The element that determines most of the motion is determined by the end point of the skeleton (the terminal node of the graph in FIG. 5), and it is defined as a key joint that determines the continuous motion.

Next, after calculating the inter-frame position change of the core joint (S152), referring to the calculated position change, core frames that distinguish human motion are selected (S153), and a time window is selected based on the selected core frames set (S154).

That is, the time window is set based on the amount of position change of the corresponding core joint (end joint of the skeleton) between each frame.

)를 계산한다. 이때, 위치 변화량(속도)이 음수 부호를 가지는 경우가 있으며 이를 제거하기 위해 유클리디언 거리 값을 사용하여 매 프레임의 사용자의 움직임 정도를 추정한다. Specifically, in order to set the time window (τ, motion section) based on the input time of the image, first, the corresponding position change (

) is calculated. At this time, there is a case where the amount of position change (velocity) has a negative sign, and in order to remove it, the degree of movement of the user in each frame is estimated using the Euclidean distance value.

When this estimated result value is 0, it means that the user's movement has stopped for a while, which can be regarded as a key frame that distinguishes a specific motion. However, since the distance difference according to the position change cannot be 0 due to the FPS of the actual input image and the noise of the 3D joint, a threshold is set and if it is lower than the threshold, it is regarded as a core frame. Then, a time window is set based on the key frames.

Next, for the frames of the time window including the current frame, time weights for each section according to the change in the distance of the core joints are combined, and the spatio-temporal characteristics are calculated by adding the temporal characteristics of the motion to the spatial characteristics of the skeleton ( S155). The specific method is as follows.

Then, from among other frames of the time window including the current frame, N frames with a large amount of change in position of the current frame and joints are extracted (S156), and spatio-temporal feature vectors of the extracted frames are combined into one to obtain a final feature vector. output (S157).

As a result, the final feature vector is a combination of spatial characteristics (distance, angle, graph structure) of the skeleton and temporal change characteristics between key joints.

Again, it will be described with reference to FIG. 2 .

After the spatio-temporal feature is calculated, the computed spatio-temporal feature is input to an artificial intelligence model (random forest, etc.) trained for human motion recognition, and human motion is inferred and recognized (S160).

8 is a block diagram of a gesture recognition system according to another embodiment of the present invention. The gesture recognition system according to an embodiment of the present invention is configured to include a camera 210 , a computing system 220 , and a display 230 , as shown.

The camera 210 is a photographing device that generates a continuous 2D image. The computing system 220 analyzes the user's movement from the image generated by the camera 210 to infer/recognize the motion. The display 230 displays an image generated by the camera 210 and an inference/recognition result from the computing system 220 .

For motion recognition, the computing system 220 detects a person in the current frame, extracts the detected joints of the person, classifies the extracted joints into a plurality of joint sets, and based on distance information between the classified joint sets Calculate the spatial features of the skeleton.

Furthermore, the computing system 220 calculates a spatio-temporal feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint, and converts the calculated spatio-temporal feature to the human body. It is input to the trained artificial intelligence model to recognize human motion by inferring and recognizing human motion.

So far, a preferred embodiment has been described in detail for a method and system for recognizing a user's motion using 3D skeleton information.

In the above embodiment, in human motion recognition, by using the spatial-temporal geometric relationship of the skeleton as a key element, by relocating the position information of the entire joint from the image center to the skeleton center, regardless of the angle and position from the camera Joint position information was used.

In addition, the physical connection relationship is obtained using distance information and angle information between joints, and the physical relationship between each joint is expressed in a graph based on the center of gravity of the skeleton, and from this, the joint set is defined and the intrinsic connection relationship between the joints is obtained. did.

And, based on the current frame, it is possible to estimate the time information of the motion by analyzing the position change and correlation between the joint sets within the time window (section).

On the other hand, it goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical ideas according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium may be any data storage device readable by the computer and capable of storing data. For example, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

210: camera
220: computing system
230: display

Claims (12)

detecting a person in the current frame;
extracting the detected human joints;
classifying the extracted joints into a plurality of joint sets;
calculating spatial characteristics of the skeleton based on distance information between the classified joint sets;
calculating a spatio-temporal feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint;
A motion recognition method comprising: inputting the calculated spatio-temporal characteristics to the artificial intelligence model trained for human motion recognition, and inferring the human motion.
The method according to claim 1,
The classification step is
rearranging the extracted joints in consideration of the physical connectivity of the extracted joints;
Classifying the relocated joints into a plurality of joint sets; motion recognition method comprising: a.
3. The method according to claim 2,
The extraction step is
Extract the 3D joint coordinates,
The classification step is
normalizing the extracted 3D joint coordinates;
Calculating the center of gravity of the skeleton from the normalized 3D joint coordinates; further comprising,
Based on the calculated center of gravity, the joints are rearranged in consideration of the physical connectivity of the joints,
A motion recognition method, characterized in that the relocated joints are classified into a plurality of joint sets based on the calculated center of gravity.
The method according to claim 1,
The step of calculating the spatial features of the skeleton is,
A method for recognizing a motion, characterized in that the spatial characteristics are calculated by reflecting the intrinsic distance information between the classified joint sets in the classified physical distance information between the classified joint sets.
5. The method according to claim 4,
physical distance information,
is the relative distance between the joints,
Intrinsic distance information is
A method for recognizing a motion, characterized in that it is determined by a relationship between joint sets including joints.
The method according to claim 1,
The spatio-temporal feature calculation step is,
A method for recognizing a motion, characterized in that the temporal-spatial characteristic is calculated by reflecting the positional change of the core joint in the spatial characteristic in the time window including the current frame.
7. The method of claim 6,
The spatio-temporal feature calculation step is,
selecting a core joint from among the extracted joints;
calculating the inter-frame position change of the core joint;
Selecting the core frames that distinguish the motion of a person with reference to the calculated position change;
Based on the selected key frames, setting a time window; Method for recognizing motion further comprising a.
8. The method of claim 7,
The selection step is
A motion recognition method, characterized in that the end joints of the skeleton are selected as core joints.
7. The method of claim 6,
The spatio-temporal feature calculation step is,
A method for recognizing motion, comprising extracting N frames in the order of the largest change in the position of key joints in a time window including the current frame, and combining the spatio-temporal features calculated using the extracted frames into one.
a camera that generates an image;
Detects a person in the current frame of the image, extracts the detected human joints, classifies the extracted joints into multiple joint sets, calculates spatial features of the skeleton based on distance information between the classified joint sets, and calculates Based on the spatial characteristics and the position change between the frames of the core joints, a spatio-temporal feature with temporal features added to the spatial features of the skeleton is calculated, and the calculated spatio-temporal features are used to recognize human motion. A motion recognition system comprising: a computing system for inferring a human motion by inputting the model.
classifying the human joints detected in the current frame into a plurality of joint sets;
calculating spatial characteristics of the skeleton based on distance information between the classified joint sets;
calculating a spatio-temporal feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint;
Inferring a human motion by using the calculated spatio-temporal features. A motion recognition method comprising: a.
classifying the human joints detected in the current frame into a plurality of joint sets;
calculating spatial characteristics of the skeleton based on distance information between the classified joint sets;
calculating a spatio-temporal feature in which a temporal feature is added to the spatial feature of the skeleton based on the calculated spatial feature and a change in position between frames of the core joint;
A computer-readable recording medium in which a program capable of performing a motion recognition method is recorded, comprising: inferring a human motion by using the calculated spatio-temporal characteristics.

Images