KR102369152B1 - Realtime Pose recognition system using artificial intelligence and recognition method - Google Patents

Abstract

According to the present invention, a real-time posture motion recognition system and a recognition method using artificial intelligence includes the steps of: acquiring an image; pre-processing the acquired image; extracting a posture by inputting the pre-processed image to a posture recognition model; and normalizing a posture recognition result and recognizing a motion by comparing a normalized posture recognition result with a reference posture. Therefore, it is possible to classify a large number of exercise motions in real time based on a posture information.

Description

Realtime Pose recognition system using artificial intelligence and recognition method

The present invention relates to a real-time posture motion recognition system and recognition method using artificial intelligence.

As indoor activities increase due to COVID-19, video content is on the rise. Accordingly, many studies are being conducted to understand, summarize, and analyze the contents of a large amount of video content. In order to analyze these numerous video contents more efficiently, deep learning technology has recently been attracting attention.

As a prior art in this regard, Korean Patent Application Laid-Open No. 2015-0079064, 'Automatic Tagging System' discloses a technique for extracting only visual physical information and semantic information of a still image, and Korean Patent Application Publication No. 2011-0020158 In 'Metadata tagging system, image search method, device and gesture tagging method applied thereto', a technique for extracting visual information and place information by analyzing an image is disclosed. However, this prior art is limited to visual information tagging within an image, and does not guarantee the quality of metadata. In addition, it is impossible to generate integrated metadata having all of visual information, sound information, subtitle information, and caption information for a single image, and tagging a large amount of data is expensive and difficult to work with.

In particular, as indoor activities increase, video-based non-face-to-face online coaching services such as online classes and home training are attracting attention. However, most video-based online coaching services are implemented as a one-way teaching method that unilaterally transfers knowledge rather than a two-way coaching method that can receive feedback. Therefore, a problem arises in that the user has to judge for himself how well he or she is doing or how consistent the results are. In particular, in the case of a home training service using video, if the content is conducted in a one-way coaching method, there is a risk of injury because there is a possibility that the user may perform an operation in the wrong way.

Therefore, in order to solve the above-mentioned problems, it is required to implement a system that analyzes the user's image, records the motion, and gives feedback. For example, information about the motion is obtained by extracting a frame in which a motion exists from within the image, and information about the motion such as the number of repetitions and similarity using the obtained motion information, and statistics on the motion for each user, etc. can provide the results of

However, in order to classify human movement movements, a machine learning model is generally configured to classify the types of movements, but there is a problem that a dataset for classification is required for each movement. In addition, there is a limitation in that it is difficult to obtain real-time performance because it takes a long time to perform such an algorithm.

Therefore, it is required to develop a method to obtain human posture information based on an image and to classify a large number of exercise movements in real time based on the obtained information.

In order to solve the above problems, the present invention is to provide a real-time posture motion recognition system and recognition method using artificial intelligence.

According to an embodiment, a method for real-time posture motion recognition executable in a computing device includes: acquiring an image; pre-processing the acquired image; extracting the posture by inputting the pre-processed image to the posture recognition model; normalizing the posture recognition results; and recognizing the motion by comparing the normalized posture recognition result with the reference posture.

The pre-processing step is

determining an inference period with a frame period; and changing the frame size to match the posture recognition model.

The posture recognition model may be a model based on deep learning.

The step of normalizing the posture recognition result is,

determining a size factor for normalization; and

and normalizing the posture recognition result value by dividing it into size factors.

The step of recognizing a motion by comparing the normalized posture recognition result and the reference posture comprises:

comparing the degree of similarity between normalized joint coordinates and a reference posture; The method may include determining that the joint coordinates are the same as the reference posture when the similarity is higher than the threshold value.

The step of recognizing a motion by comparing the normalized posture recognition result and the reference posture comprises:

defining a center point among normalized joint coordinates;

aligning the reference posture on the same line with respect to the central point; and

The method may include recognizing a motion by comparing a posture recognition result aligned on the same line and a reference posture.

The step of defining the center point may be a step of defining the joint coordinates with the least change among normalized joint coordinates as the center point.

The joint coordinates with the least change may be characterized as joint coordinates with respect to a torso.

Comparing the normalized joint coordinates and the similarity of the reference posture comprises:

It may be a step of calculating the distance-based similarity and the angle-based similarity.

Comparing the normalized joint coordinates and the similarity of the reference posture comprises:

The calculated distance-based similarity and angle-based similarity may be added with weights for each motion to derive a final similarity.

Visualizing the normalized posture recognition result and the reference posture in real time; may further include.

The visualization step is

Visualize the reference posture with a first color,

The method may include quantifying the degree of similarity between the normalized posture recognition result and the reference posture, and visualizing the result with a color other than the first color according to the similarity value.

The visualization step is

When the reference posture and the normalized posture recognition result and the degree of similarity greater than or equal to a predetermined reference value are satisfied, it is visualized in a second color, and when it is less than, it is visualized in a third color,

The method may further include determining that the reference posture is performed when the second color satisfies a ratio greater than or equal to a predetermined reference value.

Real-time posture motion recognition system according to an embodiment of the present invention,

a transceiver capable of transmitting and receiving to and from the outside through a network;

a memory unit for storing an application for performing a real-time posture motion recognition method; and

A processor that reads and controls the application from the memory unit; includes,

The application obtains an image, pre-processes the acquired image, inputs the pre-processed image to the posture recognition model to extract the posture, normalizes the posture recognition result, and compares the normalized posture recognition result with the reference posture Recognize motion.

The real-time posture motion recognition system and recognition method using artificial intelligence according to the present invention can distinguish a large number of movement movements in real time based on posture information with posture recognition technology and images using deep learning technology.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a flowchart illustrating a real-time posture motion recognition method according to the present invention.
2 is a detailed flowchart illustrating a real-time posture motion recognition method according to the present invention.
3 is a detailed flowchart illustrating a real-time posture motion recognition method according to the present invention.
4 is a detailed flowchart illustrating a real-time posture motion recognition method according to the present invention.
5 is a block diagram illustrating a real-time posture motion recognition system according to the present invention.
6 and 7 are diagrams illustrating a step of aligning joint information and a reference posture based on the center point of the real-time posture motion recognition system on the same line.
8 and 9 are diagrams specifically illustrating a visualization method of a real-time posture motion recognition method according to the present invention.
10 is a diagram illustrating an example of converting a photographing angle of a photographing apparatus into a front image by correcting the photographing angle.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Advantages and features of the present disclosure, and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the technical spirit of the present disclosure is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments complete the technical spirit of the present disclosure, and in the technical field to which the present disclosure belongs It is provided to fully inform those of ordinary skill in the art of the scope of the present disclosure, and the technical spirit of the present disclosure is only defined by the scope of the claims.

In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which this disclosure belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly defined in particular. The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present disclosure. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

In addition, in describing the components of the present disclosure, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the component from other components, and the essence, order, or order of the component is not limited by the term. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but another component is between each component. It should be understood that elements may be “connected,” “coupled,” or “connected.”

As used herein, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element of one or more other components, steps, operations and/or elements. The presence or addition is not excluded.

A component included in one embodiment and a component having a common function may be described using the same name in another embodiment. Unless otherwise stated, the descriptions in any one embodiment may be applied to other embodiments, and specific descriptions will be omitted within the overlapping range or within the range that can be clearly understood by those skilled in the art. can

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

Hereinafter, the present invention will be described in detail with reference to preferred embodiments of the present invention and the accompanying drawings.

1 is a flowchart illustrating a real-time posture motion recognition method according to the present invention. Referring to FIG. 1 , the real-time posture motion recognition method includes obtaining an image (S100), pre-processing the obtained image (S200), and inputting the pre-processed image into a posture recognition model to extract a posture (S300) ), normalizing the posture recognition result (S400) and recognizing a motion by comparing the normalized posture recognition result with the reference posture (S500).

The step of acquiring an image ( S100 ) may mean a step of acquiring all related images for the purpose of acquiring a posture of an image related to exercise such as sports analysis and personal home training. The method of acquiring an image includes a method of directly receiving a photographed image from a direct imaging device such as a camera or a smartphone, or a method of receiving a photographed image through a transceiver, and is not limited to a specific example. The image includes still image data such as a photo, moving image data continuously transmitted in a specific frame unit, and streaming moving image data being recorded in real time, and is not limited to a specific example.

The step of pre-processing the acquired image ( S200 ) is a step of pre-processing the acquired image to facilitate posture recognition. Details will be described later with reference to FIG. 2 .

The step of extracting the posture ( S300 ) is a step of obtaining a human pose estimation result by inputting the size-adjusted input image to the deep learning model. The posture extraction step according to the present invention generally obtains 17 and 34 posture estimation results, but this may vary depending on model design. In this step (S300) according to an example, a lightweight deep learning model may be used for real-time posture recognition.

The step of normalizing the posture recognition result ( S400 ) may be a step of normalizing after determining a scale factor from a part that can approximate the size of a person as a result obtained from the posture recognition result. For example, normalization not affected by the size of a person is possible by dividing the result of posture recognition into size factors. Details will be described later with reference to FIG. 4 .

The step of recognizing the motion by comparing the normalized posture recognition result with the reference posture ( S500 ) is a step of recognizing the motion by comparing the obtained joint information with the central posture. In this step (S500), angle or distance-based similarity may be used. In order to implement real-time recognition, it is possible to compare only the similarity of the joint data after determining the joint data with high importance for motion rather than comparing the similarity of all joint data. Details will be described later with reference to FIG. 4 .

2 is a detailed flowchart illustrating a real-time posture motion recognition method according to the present invention. Referring to FIG. 2 , the pre-processing includes determining an inference period as a frame period ( S210 ) and changing the frame size to match the posture recognition model ( S220 ).

The step (S210) of determining the inference period by the frame period is a frame period for improving the operation speed while maintaining the inference precision so that the real-time is not damaged due to the limitation of the operation speed when all frames of image data are inferred. (frame interval) is decided. The frame period corresponds to a kind of unit that determines how many frame periods to input to the inference model for the frame stream data input in real time.

The step of changing the frame size to match the posture recognition model ( S220 ) is a step of adjusting the input data to the frame size required by the model.

Since there is a frame input size required for each model in a deep learning model, it is necessary to adjust the input data to the frame size required by the model for real-time posture recognition. For example, when an input is received from an imaging device existing in a mobile device, an input having a specific size of A x B is received according to the specification of the device. When an input is received, a preprocessing step of selecting the received frame according to a predefined frame period and adjusting the selected frame to the size required by the deep learning model, C x D, can be performed.

3 is a detailed flowchart illustrating a real-time posture motion recognition method according to the present invention. Referring to FIG. 3 , the step of normalizing the posture recognition result includes determining a size factor for normalization ( S410 ), and normalizing the posture recognition result value by dividing it by the size factor ( S420 ).

The step of determining a size factor for normalization ( S410 ) is a step of determining a size factor capable of approximating the size of a person. The size factor can be obtained from joint information that can approximate the size of a person stably with the smallest change in position. For example, the size factor may be determined through torso joint information. This is because it is easy to approximate the size of a person because joint information about the torso can be obtained relatively stably.

The step of normalizing the posture recognition result value by dividing it into size factors ( S420 ) is a step of performing size normalization by dividing the size factor into joint information (key points).

4 is a detailed flowchart illustrating a real-time posture motion recognition method according to the present invention. Referring to FIG. 4 , the step of recognizing a motion by comparing the normalized posture recognition result and the reference posture (S500 in FIG. 1) is the step of defining a center point among the normalized joint coordinates (S510), the reference posture based on the center point aligning to the same line (S520), comparing the normalized joint coordinates with the reference posture similarity (S530), determining whether the joint coordinates are the same as the reference posture if the similarity is higher than the threshold value (540 to S550) may be included.

The step of defining the center point among the normalized joint coordinates ( S510 ) may be a step of defining the joint coordinate with the least change among the normalized joint coordinates as the center point. For example, the joint coordinates with the least change may be joint coordinates with respect to a torso.

The step of aligning the reference posture with the center point on the same line ( S520 ) is a step of aligning the joint coordinates normalized through coordinate movement and the coordinates of the reference posture with the coordinates on the same line. When the normalized joint coordinates are composed of a plurality of central joints, an alignment operation is performed to align the coordinates on the same line by moving the average value of the central joints as an axis.

Comparing the normalized joint coordinates and the similarity of the reference posture ( S530 ) may be a step of calculating the distance-based similarity and the angle-based similarity. For example, in this step ( S530 ), the calculated distance-based similarity and angle-based similarity may be added with weights for each operation to derive a final similarity.

If the degree of similarity is higher than the threshold, determining that the joint coordinates are the same posture as the reference posture includes the steps of determining the joint coordinates as the same posture as the reference posture when the similarity is higher than the threshold (S540) and the threshold If the similarity is lower than the value, determining the joint coordinates as a posture different from the reference posture (S550) may be included.

5 is a block diagram illustrating a real-time posture motion recognition system according to the present invention. Referring to FIG. 5 , the real-time posture motion recognition system 100 is read from the memory 101 , the processor 103 , the transceiver 104 , the output unit 105 , the input unit 106 , and the memory 101 . The application 102 is controlled by the processor 103 .

The processor 103 executes the overall control function of the terminal by using the program and data stored in the memory 101 configured in the terminal. The processor 103 may include a random access memory (RAM), read only memory (ROM), a central processing unit (CPU), a graphic processing unit (GPU), and a bus, and the RAM, ROM, CPU, GPU, etc. may include a bus. can be connected to each other through The processor 103 may access the storage unit to perform booting using an operating system (O/S) stored in the memory 101 , and operates as an application unit using the application 102 stored in the memory 101 . It may be configured to perform various operations described in the present invention while doing so. The processor 103 controls various components in the device of the node, that is, the memory 101, the input unit 106, the output unit 105, the transceiver unit 104, and the camera (not shown) to control various components disclosed in the present invention. may be configured to carry out embodiments.

In addition, the real-time posture recognition system 100 includes a memory 101 for storing various data including application 102-related data, an input unit 106 for receiving a user input, an output unit 105 for displaying various information, and other It may be configured to include various components such as the transceiver 104 for communication with the terminal.

The memory 101 may be composed of a database (DB) or various storage means such as a physical hard disk, a solid state drive (SSD), a web hard, and the like.

The input unit 106 and the output unit 105 may be configured as input/output units simultaneously in the form of a touch display in a smartphone. The input unit 106 may include a physical keyboard device, a touch display, an image input sensor constituting a camera, a sensor for receiving a fingerprint, a sensor for recognizing an iris, and the like. The output unit 105 may be configured as a monitor, a touch display, or the like. However, the present invention is not limited thereto, and may include a keyboard, a mouse, and a touch screen used as an input unit in a personal computer (PC), and a monitor and a speaker used as an output unit.

The transceiver 104 may include a transmitter, a receiver, or a transceiver.

In addition, this real-time posture motion recognition system 100 is a smart phone (smart phone), mobile phone, PDA (personal digital assistant), PMP (portable multimedia player), all that can be connected to an external server through a wireless communication network, such as a tablet PC. It may include a type of handheld-based wireless communication device, and in addition to this, a communication device that can be connected to an external server through a network, such as an IPTV including a desktop PC, a tablet PC, a laptop PC, and a set-top box, is also provided. may include

The application 102 acquires an image, pre-processes the acquired image, inputs the pre-processed image to a posture recognition model to extract a posture, normalizes the posture recognition result, and compares the normalized posture recognition result with a reference posture to recognize the action. Since the real-time posture motion recognition method performed by the application 102 is the same as described above with reference to FIGS. 1 to 4 , a redundant description will be omitted.

6 and 7 are diagrams illustrating a step of aligning joint information and a reference posture based on the center point of the real-time posture motion recognition system on the same line.

Referring to FIG. 6 , after changing the frame size to fit the model in a frame period (3 frames) unit determined for raw data of motion taken through a smartphone camera, the deep learning-based posture recognition model An example of extracting a posture is shown.

Referring to FIG. 7 , an example of performing normalization after determining a size factor (second image) and performing an alignment step to fit on the same line based on torso joint information (third image) is shown.

8 and 9 are diagrams specifically illustrating a visualization method of a real-time posture motion recognition method according to the present invention. 8 and 9 , the visualization information for the reference posture is shown in a first color with the joint information as the center, and the motion information obtained from the user is continuously displayed in the second to third colors based on the joint information. is shown

The user may check the reference posture motion shown on the screen and perform the motion in real time so that the motion and motion information acquired by the user match. At this time, as described above, when the degree of similarity is higher than the threshold value, determining the joint coordinates as the same posture as the reference posture (S540), and when the similarity is lower than the threshold value, determining the joint coordinates as a posture different from the reference posture In (S550), the same degree is quantified, and the color of the user joint information when the most coincident is shown as a second color, and the color of the user joint information when the degree of mismatch is greatest is shown as a third color, and the quantified According to the numerical value, it may be characterized in that the color of the user joint information is continuously determined within the range of the second color to the third color. For example, the second color may be green and the third color may be red.

If a plurality of reference postures are provided and the user needs to perform the next reference posture after forming the reference posture, if the similarity of user joint information to the provided reference posture exceeds a predetermined reference value, the reference posture is applied to the reference posture. The method according to the present invention may be provided so as to determine that the operation has been performed and to provide the next reference posture.

For example, when a reference value (eg, 70% or more) among lines connecting a plurality of joint coordinates based on a plurality of joint coordinates constituting the reference posture is shown in the first color, the user selects the reference posture It can be judged to have been properly performed. These steps may be repeatedly provided until all the prepared reference postures are provided and the user performs similar motions greater than the reference posture and the reference value.

Also, referring to FIG. 9 , an example of correcting the joint coordinate information of the reference posture to be the same as the scale of the user's joint coordinates is shown to correct the difference in the scale of the user's joint coordinates due to the distance between the user and the photographing device such as a camera.

When the reference posture coordinates visualized on the screen are always visualized with the same size, the user's joint coordinates and scales, which may be different depending on the distance between the imaging device and the user, are different, so an error may occur in performing an operation or calculating the similarity. Accordingly, the visualization of the reference posture may be corrected to change dynamically according to the distance between the photographing device (device) and the user.

Through this correction operation, the user can easily understand the result of motion recognition and can improve the precision of the feedback.

Referring to FIG. 10 , the preprocessing step ( S200 in FIG. 1 ) according to the present invention may further include converting the preprocessed images taken at various angles and positions into a frontal image.

Depending on the user's camera settings, images can be taken from various angles. For example, when a webcam is installed on a TV and viewed, an image looking down from the top rather than the front image is captured. Since images obtained from various angles can be recognized as distorted information when input to a deep learning model, preprocessing may be required.

Referring to FIG. 10 , undistorted information is obtained by converting the distorted image into a front image by re-calculating the information as a geometric function by inversely calculating the inclination angle of the photographing device based on information obtained from sensors and images. can be derived

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

Exemplary embodiments have been disclosed in the drawings and specification as described above. Although the embodiments have been described using specific terms in the present specification, these are used only for the purpose of explaining the technical spirit of the present disclosure, and not used to limit the meaning or the scope of the present disclosure described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present disclosure should be defined by the technical spirit of the appended claims.

Claims (14)

In a real-time posture motion recognition method executable on a computing device,
acquiring an image;
pre-processing the acquired image;
extracting the posture by inputting the pre-processed image to the posture recognition model;
normalizing the posture recognition results; and
Comprising the step of recognizing a motion by comparing the normalized posture recognition result and the reference posture,
In the step of normalizing the posture recognition result, a torso capable of approximating the size of a person stably with little change in position is selected as a scale factor,
In the step of recognizing the motion by comparing the normalized posture recognition result and the reference posture, the joint coordinates for the torso with the least change among the normalized joint coordinates are defined as the center point, and the reference posture is determined based on the center point. collinearity; A real-time posture motion recognition method comprising a. According to claim 1,
The pre-processing step is
determining an inference period with a frame period; and
A real-time posture motion recognition method comprising; changing a frame size to match the posture recognition model. According to claim 1,
The posture recognition model is a real-time posture motion recognition method that is a model based on deep learning. According to claim 1,
The step of normalizing the posture recognition result is,
determining a size factor for normalization; and
A real-time posture motion recognition method comprising a; normalizing the posture recognition result value by dividing it by size factors. According to claim 1,
The step of recognizing a motion by comparing the normalized posture recognition result and the reference posture comprises:
comparing the degree of similarity between normalized joint coordinates and a reference posture;
A real-time posture motion recognition method comprising the step of determining that the joint coordinates are the same as the reference posture when the similarity is higher than the threshold value. delete delete delete 6. The method of claim 5,
Comparing the normalized joint coordinates and the similarity of the reference posture comprises:
A real-time posture motion recognition method that calculates distance-based similarity and angle-based similarity. 10. The method of claim 9,
Comparing the normalized joint coordinates and the similarity of the reference posture comprises:
A real-time posture motion recognition method in which the calculated distance-based similarity and angle-based similarity are weighted for each motion and summed to derive a final similarity. According to claim 1,
Real-time posture motion recognition method further comprising; visualizing the normalized posture recognition result and the reference posture in real time. 12. The method of claim 11,
The visualization step is
Visualize the reference posture with a first color,
and quantifying the degree of similarity between the normalized posture recognition result and the reference posture, and visualizing the result with a color other than the first color according to the similarity value. 13. The method of claim 12,
The visualization step is
When the reference posture and the normalized posture recognition result and the degree of similarity above a predetermined reference value are satisfied, it is visualized in a second color, and when it is less than, it is visualized in a third color,
The method further comprising the step of determining that the reference posture is performed when the second color satisfies a ratio greater than or equal to a predetermined reference value. The real-time posture motion recognition system,
a transceiver capable of transmitting and receiving to and from the outside through a network;
a memory unit for storing an application for performing a real-time posture motion recognition method; and
A processor that reads and controls the application from the memory unit; includes,
The application is
acquire video,
Pre-process the acquired image,
The pre-processed image is input to the posture recognition model to extract the posture,
normalize the posture recognition results, and
Recognize the motion in contrast to the normalized posture recognition result and the reference posture,
When normalizing the posture recognition result, a torso capable of approximating the size of a person stably with little change in position is selected as a scale factor,
When recognizing motion in comparison with the normalized posture recognition result and the reference posture, the joint coordinates for the torso with the least change among the normalized joint coordinates are defined as the center point, and the reference posture is set on the same line based on the center point. Real-time posture motion recognition system, characterized in that it matches.

Images