KR20230073681A - Electronic device and method for motion readiness check based on joint angle change - Google Patents

Abstract

In the joint angle change-based movement readiness confirmation method according to the present disclosure, the machine learning device learns a machine learning model using a joint data set representing joint movements for each of a plurality of types of motions, and the motion recognition device performs a motion recognition process on each body Recognizing the motion of the region and collecting joint data for a corresponding joint of the body region in motion, wherein the motion analysis device receives the joint data in a streaming manner and simultaneously learns the received joint data in advance in real time. Comparing with a plurality of types of motions to determine similarity, the motion analysis device extracting one motion whose similarity is greater than or equal to a preset reference value before the motion is terminated, and the motion prediction device extracting the extracted motion It is characterized in that it includes; determining the exercise readiness state for the motion.

Description

Electronic device and method for motion readiness check based on joint angle change

The present invention relates to an electronic device and method for confirming motion readiness based on joint angle change.

BACKGROUND OF THE INVENTION As the number of people lacking in exercise increases rapidly in modern society, the frequency of diseases such as obesity and high blood pressure is steadily increasing. Accordingly, the number of people who exercise for physical strength and health management at home or at a fitness center is increasing.

Recently, portable devices that measure exercise information including movement distance, movement time, and calories consumed have been released, and users can easily measure exercise information and exercise.

If sufficient warm-up is not performed before exercise, the user may be exposed to the risk of serious injury during strength training. In addition, there are difficulties and inconveniences in determining whether or not a muscle strength exercise can be smoothly performed by a warm-up exercise.

An object of the present invention is to reduce the risk of injury and to perform continuous and efficient exercise by confirming the exercise readiness state using changes in bending angles of the knee, elbow, and neck.

In the joint angle change-based movement readiness confirmation method according to the present disclosure, the machine learning device learns a machine learning model using a joint data set representing joint movements for each of a plurality of types of motions, and the motion recognition device performs a motion recognition process on each body Recognizing the motion of the region and collecting joint data for a corresponding joint of the body region in motion, wherein the motion analysis device receives the joint data in a streaming manner and simultaneously learns the received joint data in advance in real time. Comparing with a plurality of types of motions to determine similarity, the motion analysis device extracting one motion whose similarity is greater than or equal to a preset reference value before the motion is terminated, and the motion prediction device extracting the extracted motion It is characterized in that it includes; determining the exercise readiness state for the motion.

According to an exemplary embodiment of the present invention, in order to recognize and analyze the motion of the liver, instead of using all movements of the motion and motion data for all joints, time-series acceleration-deceleration motion data for major joints selected in advance are used. By using it, it is possible to quickly check the exercise readiness state by reducing the amount of calculation and thus improving the motion estimation speed.

1 is a block diagram illustrating an electronic device 1000 for recommending an exercise event based on an exercise record.
2 is for explaining a hidden mark model (HMM).
3 is to explain a step in which the motion information generating device sets each of a plurality of types of motions before the machine learning device learns the machine learning model.
4 is a flowchart for explaining a motion prediction method according to an embodiment of the present invention.

Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that this is not intended to limit the technology described in this disclosure to specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of this disclosure. . In connection with the description of the drawings, like reference numerals may be used for like elements.

In the present disclosure, expressions such as “has,” “can have,” “includes,” or “can include” indicate the presence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as "first," "second," "first," or "second," used in the present disclosure may modify various elements regardless of order and/or importance, and may refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, a third component) do not exist between the other components.

Hereinafter, the present disclosure will be described in detail with reference to the drawings.

1 is a block diagram illustrating an electronic device 1000 for recommending an exercise event based on an exercise record.

The input/output unit 100 provides an interface with a user and/or an external device, and includes a button 110, a connector 120, a keypad 130, a display unit 140, an audio output unit 150, A vibration motor 160 and the like may be included.

The input/output unit 100 provides an input means through which a user can input data for controlling the electronic device 1000 . In addition to the above configuration, as an input means, a dome switch, a touch pad (contact capacitance method, pressure resistive film method, infrared sensing method, surface ultrasonic conduction method, integral tension measurement method, piezo effect method, etc.) ) and the like may be further provided, but is not limited thereto.

In addition, the input/output unit 100 provides an output means for outputting an audio signal, video signal, or vibration signal. When the display unit 140 and the touch pad form a layer structure to form a touch screen, the display unit 140 may be used as an input device as well as an output device. The display unit 140 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display ( 3D display) and at least one of electrophoretic display.

Under the control of the processor 600, the display unit 140 may output a screen for exercise management items necessary for the user. Under the control of the processor 600, the audio output unit 150 outputs a voice message related to the screen guided by the display unit 140, or other sound signals such as notification sounds and warning sounds according to the user's exercise state. can output The sound output unit 150 may include a speaker, a buzzer, and the like. The vibration motor 160 may output a vibration signal. For example, a vibration signal may be output together with the warning sound or instead of the warning sound. Also, the vibration motor 160 may output a vibration signal when a touch is input to the touch screen.

The sensor unit 300 may detect a state of the electronic device 1000, for example, movement or positional change, and transmit the detected information to the processor 600. Accordingly, it is possible to count the number of times a user wearing the electronic device 1000 exercises, for example, weight lifting companies or abdominal muscle exercises.

The sensor unit 300 may include a terrestrial magnetic sensor 310 , an acceleration sensor 320 , a position sensor 330 , and a gyroscope sensor 340 . In addition, sensors for detecting bio-signals of the user, such as ECG (ELECTROCARDIOGRAPHY), GSR (GALVANIC SKIN REFLEX), and pulse waves, may be provided. In addition, a temperature/humidity sensor, an infrared sensor, an air pressure sensor, a proximity sensor, and the like may be further included, but are not limited thereto.

The short-range wireless communication unit 400 provides a communication function between the electronic device 1000 and an external device, such as a fitness device or a smartphone carried by a user.

The short-range communication unit 400 includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit (WLAN) communication unit, a Zigbee communication unit, an infrared data association (IrDA) communication unit, It may include a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, etc., but is not limited thereto.

As a communication unit provided in the electronic device 1000, only the short range communication unit 400 is exemplified, but is not limited thereto, and components such as a broadcast reception unit or a mobile communication unit may be further provided.

A program for processing and controlling the processor 600 may be stored in the memory 500, and input/output data may be stored.

The memory 500 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium.

Programs stored in the memory 500 may be classified into a plurality of modules according to their functions, for example, a sensor module 510, a GPS module 520, a communication module 530, and an exercise management module 540 etc. can be classified. In addition, a UI module, a touch screen module, a notification module, a video playback module, an audio playback module, and the like may be further included according to functions to be provided by the electronic device 1000 .

The processor 600 controls the overall operation of the electronic device 1000. For example, the processor 600 may generally control the input/output unit 100, the sensor unit 300, the communication unit 400, and the like by executing programs stored in the memory 170. In addition, by executing the exercise management module 540, a management service may be provided so that the user can manage himself when performing an exercise while using the fitness equipment.

The above configuration of the electronic device 1000 is exemplified as a component required to execute the exercise management module 540 stored in the memory 500 in the processor 600 . The exercise management module 540 may be provided in an electronic device implemented in the form of a smart watch having a physiological signal monitoring function. In this case, an additional configuration suitable for the purpose, for example, a biometric authentication module, a biometric sensor, and an analysis module for analyzing a signal received from the biometric sensor may be further provided.

In the drawing, components described in each unit, that is, the input/output unit 100, the sensor unit 300, the communication unit 400, and the memory 500, are optional components, that is, one or more of them are provided. It can be.

Hereinafter, a method for personalizing and sharing an exercise program provided by the electronic device 1000 will be described in detail.

2 is for explaining a hidden mark model (HMM).

Referring to FIG. 2 , the relationship between motion and acceleration-deceleration motion data may be expressed as λ = {Adn, G, π, A, B}. Here, G is the set of hidden states, Adn is the set of observable states, π is the starting probability of the hidden state, A is the transition matrix between hidden states, and B is the probability of the output state from the hidden state. Therefore, λ can be interpreted as the probability of an operation corresponding to the learned order of Adn. When recognizing a motion, motion recognition is performed using the probability of B.

Adn={StartRS, StartRE, Slow_DownRS, Slow_DownRE, Aaa, Direction, EndRS, EndRE}

As above, Adn in Right_arm stores motion data in units of frames. For motion prediction, the present invention assumes that the current speed change is maintained based on the motion start direction of Adn and the average angular acceleration at the start of deceleration. Based on this, it is possible to predict the angle of the joint at the point at which motion is stopped or deceleration starts. Therefore, when recognizing a motion, motion prediction is performed using the predicted value without using the actual angles of EndRS and EndRE. Through this method, even if the operation is not performed to the end, it is possible to predict the end of the operation by predicting the operation in advance.

In addition to the hidden Markov model, the motion may be learned using at least one of general machine learning models such as a Gaussian mixture model.

The machine learning apparatus 120 may generate L sets of joint data sets when L (L is a positive integer) number of acceleration-deceleration motions occur to perform one motion once.

On the other hand, since the amount of data to be processed is too large when learning about all motions and all joints of the motion, and this slows down the motion prediction speed, in the present invention, each body area divided into five areas (torso, left arm) , right arm, left leg, and right leg), the movement is observed only for preselected joints, and joint data is acquired mainly for acceleration-deceleration movement, so data throughput can be reduced and motion prediction speed can be improved accordingly. .

The motion recognition device 130 recognizes the motion of each body region and collects joint data corresponding to the motion body region. The motion recognition device 130 may recognize motions of the respective body regions through at least one of, for example, a 3D camera, an ultrasonic sensor, and a kinect used in the input unit described above.

The motion recognition device 130 may convert motion images of human body joints received through the input unit into digital signals, and bundle the converted digital signals into joint data corresponding to a plurality of types of motions.

Specifically, the motion recognition device 130 determines the angle at which the starting point joint and the interlocking joint selected in advance for the body region in motion start the motion, each angle at the time of deceleration, the average angular acceleration until the start of deceleration, and the direction of movement. can be collected as joint data.

The motion analysis device 140 receives the joint data from the motion recognition device 130 in a streaming manner and compares the received joint data in real time with a plurality of types of pre-learned motions to determine a degree of similarity, and determines the degree of similarity before the motion ends. One motion whose similarity is equal to or higher than a predetermined reference value is extracted.

The issue here is before the motion analysis device 140 inputs the motion of the human body, that is, before the end information at the time when the motion ends is input, in the joint data input in the streaming method, the angles and motions of the starting joint and interlocking joint At least one information of the angle at the start of deceleration, the average angular acceleration until the start of deceleration, and the movement direction of the joint is compared with previously learned motions to determine similarity, and based on the similarity judgment, what kind of motion is determined. It is to determine if it is a motion, extract a pre-learned joint data set for the identified motion, and predict the end of the current motion based on end information included in the extracted joint data set.

The motion prediction device 150 predicts the end of the motion input through the motion recognition device 130 based on end information included in the pre-learned joint data set for the motion extracted by the motion analysis device 140. .

Specifically, the motion prediction device 150 extracts the joint data set pre-learned with respect to the motion extracted by the motion analysis device 140 . From the extracted joint data set, at least one of each angle of the origin joint and interlocking joint at the end of the motion, time information when the corresponding motion is completed, type information representing the characteristics of the corresponding motion, and location information of joints related to the corresponding motion are extracted. It is used to predict the end of an operation.

In addition, the motion prediction device 150 may calculate an end point of an input motion from a current time based on time information at which the previously learned motion is completed.

According to the motion prediction system according to an embodiment of the present invention, even if the user does not directly input a command, in the field of recognizing a specific action as a command and controlling the system, even if the user does not directly operate a device, the user's motion is analyzed and the game is played. Alternatively, in operating and controlling home appliances, reaction speed can be improved by predicting end information before a user's motion ends.

In addition, according to the motion prediction system according to an embodiment of the present invention, in order to recognize and analyze human motions, not all motions of motions and motion data for all joints are used, but time series for major joints selected in advance. By using regular acceleration-deceleration motion data, the amount of calculation can be reduced, and thus the motion prediction speed and furthermore, the reaction speed of IT technology can be improved.

Referring to FIG. 3 , a method for setting each of a plurality of types of motions by the motion information generating device will be described. That is, referring to FIG. 3 , before the machine learning device learns the machine learning model, the motion information generating device performs a step of setting each of a plurality of types of motions.

Specifically, the step of setting each motion by the motion information generating device, as shown in FIG. 3, obtaining the joint data of M (M is a positive integer, for example, 60) frames per second, N ( N is a positive integer, for example, 10, M>N) Measuring the change in position of all joints for each frame (S810), and when a specific joint moves a distance equal to or greater than a first threshold value, the corresponding frame is regarded as the starting point of the motion. Determining step (S820), continuing to record the change in position of the joint at intervals of N frames after determining the starting point (S830), and determining the motion stop point when the joint does not move beyond the second threshold. (S840) and setting the motion information of the joint recorded from the start point of the motion to the stop point as one motion (S850).

In addition, in the step of setting a plurality of types of each motion by the motion information generating device, a starting point joint serving as an origin of the motion and an interlocking joint linked to the movement of the starting point joint are set for each motion.

To this end, the motion information generating device selects a starting point joint that is the origin of motion based on the characteristics of the human model and kinematic constraints, and selects an interlocking joint that moves in conjunction with the movement of the starting point joint. Usually, the peristaltic joint corresponds to the joint adjacent to the origin joint with one node.

Here, the motion refers to a circumferential motion with a joint as the origin among human motions. Since there is a limit to the stretching of muscles between joints and adjacent joints, it is possible to grasp the movable range and position change of all joints performing circumferential motion.

In addition, the motion information generating device includes the angle of the starting joint and interlocking joint at the start of each motion, the angle at the start of deceleration during motion, the average angular acceleration until the start of deceleration, the movement direction of the starting joint and interlocking joint, The angle of the origin joint and interlocking joint at the end of the motion is included in the motion information. In addition, the motion information may further include at least one of information about when the motion is completed, type information indicating characteristics of the motion, and location information of a joint related to the motion.

On the other hand, the motion information generating device receives at least one joint motion information representing a motion state of a joint of an operating human body from an input unit. The input unit may receive motion image information of human body joints through at least one of, for example, a 3D camera, an ultrasonic sensor, and a kinect.

The motion information generating device converts the received joint motion images of the human body into digital signals, and performs a step of combining the converted digital signals into joint data corresponding to a plurality of types of motions.

4 is a flowchart for explaining a motion prediction method according to an embodiment of the present invention.

Referring to FIG. 4 , in the motion prediction method according to an embodiment of the present invention, first, the machine learning device learns a machine learning model using a joint data set representing motions of joints for each of a plurality of types of motions (S910).

To this end, the machine learning apparatus generates a joint data set including information about time-sequential acceleration-deceleration motions of the origin joint, which is the origin of the motion, and the interlocking joint linked to the movement of the origin joint, for one motion. Then, in order to learn a plurality of types of motions, joint data sets are classified for each motion, and a machine learning model for a plurality of types of motions is learned using the joint data sets.

Here, the machine learning model may learn an operation using at least one of general machine learning models such as a hidden mark model (HMM) and a Gaussian mixture model.

The machine learning apparatus may generate L sets of joint data sets when L (L is a positive integer) number of acceleration-deceleration motions occur to perform one motion once. In this way, time-sequential acceleration-deceleration motion data is generated.

Next, the motion recognition device recognizes the motion of each body region and collects joint data for the corresponding joint of the body region in operation (S920). To this end, the motion recognition device may recognize motions of the respective body regions through at least one of, for example, a 3D camera, an ultrasonic sensor, and a kinect used in the input unit described above.

The motion recognition device may convert motion images of human joints received through the input unit into digital signals, and bundle the converted digital signals into joint data corresponding to a plurality of types of motions.

Specifically, the motion recognition device converts each angle at the start of motion, each angle at deceleration, the average angular acceleration until the start of deceleration, and the direction of movement of the starting point joint and interlocking joint selected in advance for the body region in motion as joint data. collect as

Next, the motion analysis device receives the joint data in a streaming manner and compares the received joint data in real time with a plurality of previously learned motions to determine similarity (S930), and the motion analysis device ends the motion. One motion corresponding to a similarity equal to or higher than a predetermined reference value is extracted (S940).

Specifically, the motion analysis device receives joint data from the motion recognition device in a streaming manner, compares the received joint data in real time with a plurality of previously learned motions to determine similarity, and determines the similarity before the motion ends. One motion corresponding to a predetermined reference value or higher is extracted.

The issue here is before the motion analysis device inputs the motion of the human body, that is, before the end information at the time when the motion ends is input, in the joint data input in the streaming method, the angle of the origin joint and interlocking joint, and the deceleration during motion The degree of similarity is determined by comparing information of at least one of the angle at the start, the average angular acceleration until the start of deceleration, and the movement direction of the joint with previously learned motions, and based on the similarity judgment, it is determined what type of motion the motion is. And, by extracting a pre-learned joint data set for the identified motion, the end of the current motion can be predicted based on end information included in the extracted joint data set.

Next, the motion prediction device predicts the end of the motion based on the end information included in the pre-learned joint data set for the extracted motion (S950).

Specifically, the motion prediction device extracts a joint data set previously learned for the motion extracted by the motion analysis device. From the extracted joint data set, at least one of each angle of the origin joint and interlocking joint at the end of the motion, time information when the corresponding motion is completed, type information representing the characteristics of the corresponding motion, and location information of joints related to the corresponding motion are extracted. It is used to predict the end of an operation.

In addition, the motion prediction device may calculate an end point of an input motion from a current time based on time information at which the previously learned motion is completed.

According to the motion prediction method according to an embodiment of the present invention, even if the user does not directly input a command, the user's motion is analyzed to play a game even if the user does not directly operate a device in the field of recognizing a specific action as a command and controlling the system. Alternatively, in operating and controlling home appliances, reaction speed can be improved by predicting end information before a user's motion ends.

In addition, according to the motion prediction method according to an embodiment of the present invention, in order to recognize and analyze human motions, not all motions of motions and motion data for all joints are used, but time series for major joints selected in advance. By using regular acceleration-deceleration motion data, the amount of calculation can be reduced, and thus the motion prediction speed and furthermore, the reaction speed of IT technology can be improved.

In addition, the embodiments of the present invention can be implemented as computer readable codes in a computer readable recording medium. The computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention belongs.

A method and system for recognizing human behavior according to unit motions of the present invention extracts meaningful main unit motions from among human joint motion information, and hierarchically classifies the extracted main unit motions according to at least one information among type, time, and location. By classifying into, there is an effect of improving the accuracy of human action recognition through joint movement.

In addition, the method and system for recognizing human behavior according to unit motions of the present invention significantly reduce the amount of data computation required for human behavior recognition by detecting motion features from meaningful main unit motions excluding meaningless noise data among human motions. This has the effect of shortening the time required for recognizing human behavior.

In the above, the applicant has described the preferred embodiments of the present invention, but these embodiments are only one embodiment of implementing the technical idea of the present invention, and any changes or modifications are the same as those of the present invention as long as they implement the technical idea of the present invention. should be construed as falling within the scope.

Claims (4)

Learning, by a machine learning device, a machine learning model using a joint data set representing movement of a joint for each of a plurality of types of motion;
recognizing, by a motion recognition device, a motion of each body region, and collecting joint data for a corresponding joint of the body region in motion;
A motion analysis device receiving the joint data in a streaming manner and comparing the received joint data with the plurality of previously learned motions in real time to determine a degree of similarity;
extracting, by the motion analysis device, one motion corresponding to a predetermined reference value or higher in similarity before the motion ends; and
A method for confirming motion readiness based on a change in joint angle, comprising the step of determining, by a motion prediction device, a motion readiness for the extracted motion. According to claim 1,
The joint angle change-based exercise readiness confirmation method further comprising the step of setting each of the plurality of types of motion prior to the step of learning the machine learning model by the machine learning device. According to claim 2,
A method for confirming readiness for motion based on a change in joint angle, further comprising setting a starting point joint serving as the origin of the motion for each motion and an interlocking joint interlocking with the movement of the starting point joint. According to claim 3,
For each of the motions, the angle of the origin joint and interlocking joint at the start of the motion, the angle at the start of deceleration during motion, the average angular acceleration until the start of the deceleration, the direction of movement, and the origin joint at the end of the motion and an angle of an interlocking joint is included in the motion information.

Images