WO2023106846A1 - Skeleton point-based service providing apparatus and method - Google Patents

Abstract

Disclosed are a skeleton point-based service providing apparatus and method. The skeleton point-based service providing apparatus comprises: an input/output unit that detects movement of a user; and a control unit that extracts skeleton points that are positional information for the joints of the body from the detected movement of the user, stores the skeleton points in a memory, calculates eigen vectors on the basis of the skeleton points, and detects a postural change from the movement of the user.

Description

Skeleton point based service providing device and method

Embodiments disclosed in this specification relate to an apparatus and method for providing a service based on skeleton points, and more specifically, to store only skeleton points, which are positional information of joints of a user's body performing an exercise, and to provide services using the skeleton points. It relates to an apparatus and method for providing a service based on a skeleton point that can be provided.

This application claims priority based on December 09, 2021 and December 07, 2022, and all contents disclosed in the specification and drawings of the application are incorporated into this application.

In recent years, many people are interested in health and diet. Accordingly, interest in exercise also tends to increase.

However, it is important to exercise steadily, but it is very important to perform the exercise in the correct posture to prevent injury.

Accordingly, many people want to learn exercise through personal training (hereinafter referred to as PT), but PT is expensive and has a problem in that location is limited. In addition, due to various viruses, the number of users who want to exercise non-face-to-face is increasing.

Accordingly, a technique for analyzing an image of a user's movement and providing a feedback service to the user based on the analyzed content is being developed.

However, in the prior art as described above, since the capacity of the image used for analysis is large, there is a problem in that the capacity of the server, which is a space for storing it, must be sufficiently secured. In addition, when a user's video is used as it is, if the corresponding video is leaked due to hacking of a server by a hacker, problems such as invasion of privacy may occur.

On the other hand, the above-mentioned background art is technical information that the inventor possessed for derivation of the present invention or acquired in the process of derivation of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention. .

(Patent Document 1) Korean Patent Registration No. 10-1989447 (Announced on June 14, 2019)

Embodiments disclosed in this specification are aimed at providing a skeleton point-based service providing apparatus and method capable of storing only skeleton points, which are positional information of a user's body joint performing an exercise, and providing a service using the stored skeleton points. there is.

Other objects and advantages of the present invention may be understood from the following description, and will be more clearly understood by an embodiment. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations thereof set forth in the claims.

As a technical means for achieving the above technical problem, an apparatus for providing a skeleton point-based service includes an input/output unit for detecting a user's movement operation; And from the detected movement of the user, a skeleton point, which is position information of body joints, is extracted and stored in a memory, and an eigen vector is calculated based on the skeleton point, so that in the movement of the user, the user It includes; a control unit for detecting a change in the movement posture of the.

According to another embodiment, a service providing method performed by a skeleton point based service providing apparatus may include detecting a motion of a user; extracting a skeleton point, which is position information of a body joint, from the detected motion of the user and storing it in a memory; and calculating an eigen vector based on the skeleton point, and detecting a change in the user's movement posture in the movement motion of the user.

According to another embodiment, the recording medium is a computer readable recording medium on which a program for performing a service providing method is recorded. The service providing method performed by the skeleton point based service providing apparatus may include: detecting a motion of a user; extracting a skeleton point, which is position information of a body joint, from the detected motion of the user and storing it in a memory; and calculating an eigen vector based on the skeleton point, and detecting a change in the user's movement posture in the movement motion of the user.

According to another embodiment, the computer program is a computer program that is executed by a skeleton point-based service providing device and stored in a recording medium to perform a service providing method. The service providing method performed by the skeleton point based service providing apparatus may include: detecting a motion of a user; extracting a skeleton point, which is position information of a body joint, from the detected motion of the user and storing it in a memory; and calculating an eigen vector based on the skeleton point, and detecting a change in the user's movement posture in the movement motion of the user.

According to any one of the above-described problem solving means, rather than storing and utilizing a large-capacity motion image for analyzing a user's motion, only a skeleton point, which is positional information of a user's body joint with a relatively small capacity than an image, is stored. Since the service is provided by using the storage space, there is an advantage in that the storage space can be used more efficiently.

In addition, according to any one of the above-described problem solving means, instead of storing and using an image including the user's face for analyzing the user's motion, only skeleton points, which are position information of the user's body joints, are stored, and this is Since the service is provided by utilizing the storage space, there is an effect that even if the server, which is a storage space, is hacked by a hacker, the privacy of the user can be protected without being violated.

Effects obtainable from the disclosed embodiments are not limited to those mentioned above, and other effects not mentioned are clear to those skilled in the art from the description below to which the disclosed embodiments belong. will be understandable.

Hereinafter, the accompanying drawings illustrate preferred embodiments disclosed in this specification, and serve to further understand the technical idea disclosed in this specification together with specific details for carrying out the invention. The contents should not be construed as limited only to the matters described in such drawings.

1 is a functional block diagram of a skeleton point based service providing apparatus according to an embodiment.

2 is an exemplary diagram for explaining an apparatus for providing a service based on a skeleton point according to an exemplary embodiment.

3 is a flowchart of a service providing method according to an exemplary embodiment.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. Embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the characteristics of the embodiments, detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong are omitted. And, in the drawings, parts irrelevant to the description of the embodiments are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a component is said to be “connected” to another component, this includes not only the case of being “directly connected” but also the case of being “connected with another component intervening therebetween”. In addition, when a certain component "includes" a certain component, this means that other components may be further included without excluding other components unless otherwise specified.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a functional block diagram of an apparatus for providing a service based on a skeleton point according to an embodiment, and FIG. 2 is an exemplary view illustrating an apparatus for providing a service based on a skeleton point according to an embodiment.

The skeleton point-based service providing apparatus 100 according to the present embodiment may be implemented as an electronic terminal in which an application capable of interacting with a user is installed, may be implemented as a server, or may be implemented as a server-client system, and may be implemented as a server-client system. If it is, it may include an electronic terminal in which an application for an online service for interaction with a user is installed.

At this time, the electronic terminal may be implemented as a computer, portable terminal, television, wearable device, etc. capable of accessing a remote server through a network or connecting to other devices and servers. Here, the computer includes, for example, a laptop, desktop, or laptop equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , PCS(Personal Communication System), PDC(Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), GSM(Global System for Mobile communications), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), Smart Phone, Mobile WiMAX (Mobile Worldwide Interoperability for Microwave Access), etc. (Handheld)-based wireless communication device may be included. In addition, television may include IPTV (Internet Protocol Television), Internet TV (Internet Television), terrestrial TV, cable TV, and the like. Furthermore, a wearable device is a type of information processing device that can be worn directly on the human body, such as, for example, a watch, glasses, accessories, clothes, shoes, etc. can be connected with

In addition, the server may be implemented as a computer capable of communicating over a network with an electronic terminal having an application for interaction with a user of the skeleton point-based service providing device 100 or a web browser, or may be implemented as a cloud computing server. In addition, the server may include a storage device capable of storing data or may store data through a third server.

As described above, the skeleton point-based service providing device 100 may be implemented in any one form of an electronic terminal, a server, or a server-client system, and when implemented as a server, the skeleton point-based service providing device 100 The constituting unit may be performed on a plurality of physically separated servers or on one server.

Referring to FIG. 1 , an apparatus 100 for providing a service based on a skeleton point according to an embodiment includes an input/output unit 110, a communication unit 120, a memory 130, and a control unit 140.

The input/output unit 110 may include an input unit for receiving an input from a user and an output unit for displaying information such as a result of performing a task or a state of the skeleton point based service providing apparatus 100 . For example, the input/output unit 110 may include an operation panel for receiving a user input and a display panel for displaying a screen. According to an embodiment, the input/output unit 110 may detect a user's motion. For example, the input/output unit 100 may be a motion camera capable of photographing a user's movement and action, and through this, the user's movement and action may be photographed and sensed.

Specifically, the input unit may include devices capable of receiving various types of inputs, such as a keyboard, a physical button, a touch screen, a camera, or a microphone. Also, the output unit may include a display panel or a speaker. However, the input/output unit 110 is not limited thereto and may include a configuration supporting various input/outputs.

The communication unit 120 may perform wired/wireless communication with other devices (devices) and/or networks. To this end, the communication unit 120 may include a communication module supporting at least one of various wired/wireless communication methods. For example, the communication module may be implemented in the form of a chipset.

Meanwhile, wireless communication supported by the communication unit 120 includes, for example, Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, Bluetooth Low Energy (BLE), and Ultra Wide Band (UWB). , Near Field Communication (NFC), LTE, LTE-Advanced, and the like. In addition, wired communication supported by the communication unit 120 may be, for example, USB or High Definition Multimedia Interface (HDMI).

The memory 130 may install and store various types of data such as files, applications, and programs, and may include at least one of various types of memories such as RAM, HDD, and SSD. The controller 140 to be described later may access and use data stored in the memory 130 or may store new data in the memory 130 . Also, the controller 140 may execute a program installed in the memory 130 . Meanwhile, a program for performing a service providing method according to an embodiment may be installed in the memory 130 . According to an embodiment, the memory 130 may store the skeleton points extracted by the controller 140 to be described later. In this case, the skeleton point may mean location information of body joints. In this case, the control unit 140 may extract a skeleton point composed of joints and bones based on the skeleton from the motion of the user detected by the input/output unit 100 . For example, a skeleton point is location information of joints such as the neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, etc. It may include, and may be displayed as 2D or 3D coordinate information. According to an embodiment, since a user's motion is captured by a portable terminal such as a mobile phone, it may be information included in a 2D image frame. Therefore, the skeleton point according to an embodiment is preferably displayed as 2D coordinate information. Accordingly, the present invention according to an embodiment may estimate a directionality by calculating an eigenvector based on a skeleton point indicated by 2D coordinate information, and a more detailed description regarding this will be described later. Meanwhile, the skeleton points according to an embodiment may be stored for each time. That is, since the motion of the user changes based on time-series characteristics, the motion motion of the user can be detected in more detail by storing the skeleton points by time as described above.

The control unit 140 has a configuration including at least one processor such as a CPU, GPU, Arduino, and the like, and may control the overall operation of the skeleton point based service providing apparatus 100 . That is, the control unit 140 may control other elements included in the skeleton point-based service providing apparatus 100 to perform an operation for providing a service. Also, the controller 140 may execute a program stored in the memory 130, read a file stored in the memory 130, or store a new file in the memory 130.

According to the embodiment, the control unit 140 may extract a skeleton point, which is position information of body joints, from the user's motion detected through the input/output unit 120 and store it in the memory 130 . More specifically, the controller 140 recognizes the user's skeleton from the user's motion, extracts joints and bones based on the skeleton, and then extracts spatial coordinate values for the joints from the extracted information, thereby obtaining a skeleton point. can be extracted. In this case, the skeleton point may be two-dimensional coordinate information, and may be displayed as (x, y) values, for example. At this time, the skeleton point including the 2D coordinate information is based on a point in one image frame including the user's motion (eg, a point when the upper left corner of the frame is set to (0, 0)) As a result, the x-axis value and the y-axis value can be extracted. Also, the skeleton point may be 3D coordinate information, and may be displayed as (x, y, z) values, for example. In this case, the extraction of the skeleton point from the user's motion may be performed through a pre-learned artificial intelligence model. For example, a pretrained artificial intelligence model for extracting a skeleton point may use an OpenPose-based algorithm. An artificial intelligence model using an OpenPose-based algorithm can recognize a person from a user-detected image and extract a skeleton point as the recognized person moves. In this case, the OpenPose-based algorithm may be a known joint detection artificial intelligence algorithm that detects a moving user's posture in real time using multi-GPU-based deep learning and analyzes the user's movement. In this case, the detected joint may be the neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, etc. Joint position information including 2-dimensional or 3-dimensional coordinate information for may be extracted as a skeleton point.

Based on the skeleton points stored in the memory 130, the controller 140 may detect a change in the user's movement posture from the user's movement using a preset eigen vector. In this case, the user's movement posture change may indicate, for example, whether the user's movement is upward or downward, and whether the movement posture change means a large motion or a small motion. In this case, the eigen vector may be an eigen value capable of obtaining a direction in which the collected data coordinates form on the coordinate plane. Accordingly, when the controller 140 detects the user's motion using a preset eigen vector at the skeleton point, it can detect a change in the user's motion posture. On the other hand, since the coordinates included in the above-described skeleton points are not physical distances but pixel coordinates in a frame including the user's movement, a normalization process may be required. Based on the skeleton points including the normalized coordinates, A change in the user's movement posture may be detected. That is, in the normalization process, for example, when a person standing close to the camera exercises and a person standing far from the camera exercises, even though the size and direction of the motion are actually the same, the user's movement posture is determined using the eigenvector. In the case of calculating the change, it is necessary to make the size of the user's movement constant, as the change in coordinates of a person standing close to the camera is calculated much larger. That is, the normalization process may be performed to similarly calculate the change in the user's movement posture regardless of the perspective of the camera that has photographed the user.

At this time, the above-described normalization process may be as follows.

Set the upper left corner to (0,0) based on where the user is standing, and normalize it by changing the starting points of x and y coordinates. That is, the upper left coordinates (A in FIG. 2) of the image frame including the user performing the motion operation can be set to (0,0), which is the coordinates of the upper left corner (B in FIG. 2) based on the user standing. It can be normalized by setting to (0,0) and changing the starting point of x and y coordinates. On the other hand, the starting point is to set the upper left corner of the person standing at (0,0), divide the corresponding coordinates using the width and width, and use the x and y values relative to the width or height. there is. At this time, the starting point for normalization can be calculated according to the formula below.

· Starting point of x coordinate: (center of hip) - (shoulder width / 2)

· Starting point of y-coordinate: (center of hip) - (shoulder width)

Width: shoulder width

· Height: shoulder width * 2

Meanwhile, based on the starting point of the coordinates calculated as described above, the 2D coordinates included in the normalized skeleton points may be converted according to the formula below.

x ⇒ (initial x coordinate - x coordinate starting point) / width ＊ 100

y ⇒ (initial y coordinate - y coordinate starting point) / width ＊ 100

Meanwhile, the controller 140 detects a change in the user's movement posture based on the skeleton point including the coordinates that have undergone the normalization process described above, but may also detect the change in the user's movement posture through the following process.

First, the controller 140 detects a change in the user's movement posture based on a skeleton point including coordinates that have undergone the above-described normalization process, but the skeleton point used to detect the change in movement posture is based on the coordinates that have undergone the normalization process. Data prior to the time when the included skeleton point is stored may be collected to detect a change in the user's movement posture. For example, if a skeleton point including normalized coordinates is stored at 11:11:11 on November 11, 2022, the skeleton points collected to detect a change in the user's movement posture are at November 11, 2022. It may be data up to 11:11:10 of the day. At this time, the shape of the skeleton points collected to detect the change in the user's movement posture can be represented by the following matrix.

On the other hand, if such a matrix is substituted into the root formula, two directions, positive and negative, may appear. At this time, the user's movement direction may be determined according to the calculation result by substituting the matrices collected over time into the root formula as follows.

[Equation by substituting the collected matrices into the root formula]

For example, by substituting a matrix composed of data collected at time t-2 and a matrix composed of data collected at time t-1 into the root formula, a negative number is obtained, and a matrix composed of data collected at time t-1 and If the matrix composed of the data collected at time t also has a negative number, it may be determined that the user's movement is in the same direction. On the other hand, that is, by substituting a matrix composed of data collected at time t-2 and a matrix composed of data collected at time t-1 into the root formula, a negative number is obtained, and a matrix composed of data collected at time t-1 and If the matrix composed of the data collected at time t also has a positive number, it may be determined that the user's movement is in a different direction.

Thereafter, the controller 140 may obtain eigenvectors at each joint by using the collected skeleton points over time. More specifically, a matrix of each joint including the collected skeleton points is calculated as follows and expressed as a covariance matrix, and eigenvectors and eigenvalues may be calculated by calculating the covariance matrix as follows. In this case, the eigenvector may represent the user's movement direction, and the eigenvalue may represent the user's movement magnitude.

[Covariance matrix formula]

[Calculation formula for calculating eigenvectors and eigenvalues (λ)]

Thereafter, the controller 140 may detect a change in the user's movement posture by smoothing the eigenvectors obtained at each joint. Meanwhile, smoothing of eigenvectors can be performed by the following formula.

[Eigenvector smoothing formula]

Here, t is the time (time point), T is the last time (time point), e _t is the prediction error within the sample one step ahead, and y _t may mean the sample.

As such, upon detecting a change in the user's movement posture, the controller 140 may perform a subsequent process such as calorie calculation, which will be described later.

Meanwhile, the controller 140 may determine whether the user's movement is in an exercise state by using a preset artificial intelligence technique based on a skeleton point. At this time, the preset artificial intelligence technique may be a Long Short Term Memory (LSTM) model. An LSTM (Long Short Term Memory) model may be an artificial intelligence model used for natural language processing. More specifically, LSTM is a model derived from a recurrent neural network (RNN), and may be configured in such a way that an output value of a previous model enters an input value of a next model. At this time, the data input to the LSTM model is a skeleton point collected at regular time intervals, not every time interval, by treating all joint position coordinates from a certain time ago to the current point in time as one instantaneous serial data. can As described above, the reason why skeleton points are collected at regular time intervals is that, when collected at every time interval, only positional coordinates of joints corresponding to similar postures are included in one instantaneous serial data. Therefore, in order to grasp the movement flow of the user, the skeleton points are collected at regular time intervals, and the positional coordinates of joints corresponding to different postures can be included in one instantaneous serial data. Thus, the data input to the LSTM model It is possible to determine whether the user's motion is in an exercise state or not through

Meanwhile, when learning the above-described LSTM model, as the performance of the device equipped with the LSTM model is different, the accuracy may also be different. Therefore, no matter what device the LSTM model is loaded on, in order to be able to calculate similar accuracy performance, when learning an LSTM model, the number of input values is reduced by a certain offset from the maximum number By inserting the data together and learning by inserting a value of 0 for the remaining insufficient number using zero padding, similar accuracy can be calculated even if various numbers of values are input. According to the embodiment, it is possible to verify whether the motion of the user is in an exercise state by using the LSTM model as described above. That is, it is possible to detect a change in the user's movement posture by calculating an eigenvector based on a skeleton point extracted from the user's movement motion. At this time, the skeleton point used in the calculation of the eigenvector is input to the LSTM model. By doing so, it is possible to verify whether the result calculated based on the eigenvector is correct, that is, whether the user's motion is correct. For example, only when a change in the user's movement posture detected by calculating the eigenvector matches a change in the user's movement posture detected using the LSTM model, the calorie calculation operation described below may be performed.

In addition, the controller 140 may calculate calories according to a change in the user's movement posture based on Equation 1 below.

[Equation 1]

Here, result represents the amount of calories consumed according to the change in the user's movement posture, and O represents the amount of oxygen consumption. In addition, β represents the amount of oxygen consumption (MET) used by a body weight of 1 kg for 1 minute, w is mass, α is gravitational acceleration, and t can represent time. On the other hand, the oxygen consumption (MET) may be expressed as a value obtained by multiplying the eigen value (λ), e, calculated as described above by the skeletal weight (w), which is summed by the total amount of oxygen consumed during movement. It could be. In this case, the weight of the skeleton is given by varying specific gravity according to the size of the skeleton, and may be set between 0 and 1 in units of 0.1. In addition, in calculating α, g may indicate gravitational acceleration, w may indicate a skeletal weight, and g' may indicate a direction in which a joint moves based on a direction of gravity.

Meanwhile, the calories calculated as described above may be provided to the user.

On the other hand, the calorie calculated as described above may be compared with the calorie calculated by the device that calculates the calorie according to a preset algorithm, and if the comparison result is out of the preset range, it is adjusted according to the more accurately calculated calorie. You can also correct the value. In this case, the device that calculates calories according to a preset algorithm may be a smart watch worn by the user, but is not limited thereto, and any device capable of calculating calories may be used. At this time, the smart watch is excellent in accuracy as it calculates calories by considering complex factors such as the number of steps, heart rate, and body age. Therefore, when the calorie calculated as described above is compared with the calorie calculated by the smart watch, if the comparison result is out of a preset range, the value may be corrected with the calorie calculated by the smart watch, and the correction Numerical values may be stored in the memory 130. For example, when the calorie calculated according to Equation 1 is 100 kcal and the calorie calculated by the smart watch is 120 kcal, a weight of 1.2 times may be set as a correction value. As described above, the correction value is applied to the calorie calculated according to Equation 1 when a similar exercise is performed even if the user later exercises without wearing a smart watch, so that the user can obtain more accurate calorie information. . In addition, the correction value according to the above description may also be utilized when correcting calorie information of other users. Therefore, for example, if a 1.2-fold correction value is set for 'squat' and the calories consumed by other users who have performed 'squat' exercise are calculated as 200 kcal according to Equation 1, the control unit 140 finally By applying a correction value of 1.2 to the calories consumed by other users, 240 kcal can be output as calorie information.

As described above, the change in the user's movement posture is not detected through the difference in inclination and size by comparing the image frames including the user's movement in the order of the front and back arranged according to the time sequence, but extracted It is to detect based on the skeleton point. Accordingly, there is an advantage in that the storage space can be utilized more efficiently by storing only the skeleton points, which are positional information of the joints of the user's body, which have a relatively smaller capacity than the image, and providing a service using the skeleton points. In addition, since an image containing the user's face is not stored and used to analyze the user's movement, only the skeleton point, which is the location information of the user's body joints, is stored and the service is provided using this, Even if the server, which is a storage space, is hacked, there is an effect that the user's privacy can be protected without being violated. In addition, according to the foregoing, the present invention calculates an eigenvector and determines the direction of the calculated eigenvector, detects a change in the user's movement posture in the user's movement motion, so that the user can exercise (for example, squat or lettuce). Since it is simply determined whether the user's movement posture change (movement of the arm) is correctly performed, such as pull-down, the resources used to detect the user's movement posture change can be minimized. According to an embodiment, the controller 140 calculates a skeleton point for a predetermined exercise and detects a change in movement posture by considering the angle between each joint, but for another predetermined exercise, the embodiment disclosed herein Resources required for operation of the service providing apparatus 100 can be saved by detecting only a change in the movement posture according to the .

According to an embodiment, the controller 140 may create an avatar. At this time, the avatar may be a virtual character, and the controller 140 obtains body information including any one of body shape and height based on the user's skeleton points, and generates an avatar corresponding to the obtained body information. can In this case, various types of avatars based on body information including a skeleton point may be stored in the memory 130 . For example, if the y1 and y2 coordinates included in the user's skeleton point are 1 and 5, since y represents the measured distance in terms of height, Since body information can be acquired, the controller 130 can select and create an avatar corresponding to the acquired body information from the memory 130 . In addition, since avatars stored in the memory 130 have body information including skeleton points, the controller 140 may select and create an avatar having body information corresponding to the user's skeleton points from the memory 130. there is. On the other hand, according to an embodiment, the avatar is a virtual character implemented as a graphic image based on a skeleton point, and according to the above, since the avatar is generated to correspond to body information obtained based on the user's skeleton point, the controller ( 140) may control the avatar to correspond to the user's movement based on the skeleton point by applying the user's skeleton point to the avatar.

Also, according to an embodiment, the controller 140 may compare the motion of the avatar generated as described above with the motion of a preset model, and provide a service according to the comparison result. In addition, the control unit 140 may compare the motion of the user and the motion of a preset model based on the skeleton points, and provide a service according to the comparison result (eg, a service of providing feedback information related to motions that do not match). there is. At this time, the preset model may be a person in the video performing the same motion as the avatar's motion, or a person in the video performing the same motion as the user's motion when the user performs the motion, for example. , If the exercise performed by the avatar or user is yoga, it may be an influencer with a reputation as a yoga expert. On the other hand, the controller 140 corrects the avatar to correspond to a preset model based on body information such as body shape and/or height, and corrects the corrected avatar to match the preset model before comparing the avatar's motion with the preset model's motion. It can be overlaid on a model to compare its behavior. In this case, correcting the avatar to correspond to a preset model may be performed according to a preset body correction algorithm. As described above, the controller 140 may compare the motion of the generated avatar with the motion of the preset model, and if the motion of the avatar does not match the motion of the preset model, a warning sound is sent to inform the user that the motion is incorrect. can be made aware of. Accordingly, the user can further maximize the exercise effect by concentrating more on the motion of the preset model and having the correct exercise posture.

On the other hand, the controller 140 compares the motion of the generated avatar with the motion of a preset model, and if the motion of the avatar is inconsistent with the motion of the preset model, the controller 140 stores the inconsistent motion in the memory 130, and then the user When all of the exercise motions of are completed, a feedback service showing correct motions together with inconsistent motions may be provided to the user. Accordingly, the user can further maximize the exercise effect by confirming his/her wrong exercise motion and repeatedly mastering the correct exercise motion corresponding thereto. On the other hand, if the motion of the avatar does not match the motion of the preset model, it may be different from the calories calculated when the correct motion is performed. Therefore, when the motion of the avatar is inconsistent with the motion of the preset model, the final calorie according to the motion of the user is calculated using the following equation, so that more accurate calorie can be calculated and correct information can be provided to the user. .

<Equation 2>

Final calorie = Calorie calculated according to Equation 1 * Concordance rate

Concordance rate = (total number of motions - number of discrepant motions) / total number of motions

3 is a flowchart of a service providing method according to an exemplary embodiment.

The service providing method according to the embodiment shown in FIG. 3 includes steps processed time-sequentially in the skeleton point based service providing apparatus 100 shown in FIGS. 1 and 2 . Therefore, even if the contents are omitted below, the contents described above with respect to the skeleton point-based service providing apparatus 100 shown in FIGS. 1 and 2 can be applied to the service providing method according to the embodiment shown in FIG. 3 there is.

Referring to FIG. 3 , the skeleton point-based service providing apparatus 100 according to an embodiment may detect a user's movement operation (S310).

Next, the skeleton point-based service providing apparatus 100 may extract, from the user's movement motion detected in step S310, skeleton points, which are positional information of body joints, and store them in a memory (S320).

Next, the skeleton point-based service providing apparatus 100 calculates an eigen vector based on the skeleton point, and may detect a change in the user's movement posture from the user's movement motion (S330).

The term '~unit' used in the above embodiments means software or a hardware component such as a field programmable gate array (FPGA) or ASIC, and '~unit' performs certain roles. However, '~ part' is not limited to software or hardware. '~bu' may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program patent code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables.

Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or separated from additional components and '~units'.

In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

Meanwhile, the service providing method according to an embodiment described through this specification may be implemented in the form of a computer-readable medium storing instructions and data executable by a computer. In this case, instructions and data may be stored in the form of program codes, and when executed by a processor, a predetermined program module may be generated to perform a predetermined operation. Also, computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, a computer-readable medium may be a computer recording medium, which is a volatile and non-volatile memory implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It can include both volatile, removable and non-removable media. For example, the computer recording medium may be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD, and Blu-ray disc, or a memory included in a server accessible through a network.

In addition, the service providing method according to one embodiment described through this specification may be implemented as a computer program (or computer program product) including instructions executable by a computer. A computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, object-oriented programming language, assembly language, or machine language. . Also, the computer program may be recorded on a tangible computer-readable recording medium (eg, a memory, a hard disk, a magnetic/optical medium, or a solid-state drive (SSD)).

Accordingly, the service providing method according to an embodiment described through this specification may be implemented by executing the computer program as described above by a computing device. A computing device may include at least some of a processor, a memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components are connected to each other using various buses and may be mounted on a common motherboard or mounted in any other suitable manner.

Here, the processor may process commands within the computing device, for example, to display graphic information for providing a GUI (Graphic User Interface) on an external input/output device, such as a display connected to a high-speed interface. Examples include instructions stored in memory or storage devices. As another example, multiple processors and/or multiple buses may be used along with multiple memories and memory types as appropriate. Also, the processor may be implemented as a chipset comprising chips including a plurality of independent analog and/or digital processors.

Memory also stores information within the computing device. In one example, the memory may consist of a volatile memory unit or a collection thereof. As another example, the memory may be composed of a non-volatile memory unit or a collection thereof. Memory may also be another form of computer readable medium, such as, for example, a magnetic or optical disk.

And, the storage device may provide a large amount of storage space to the computing device. A storage device may be a computer-readable medium or a component that includes such a medium, and may include, for example, devices in a storage area network (SAN) or other components, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory, or other semiconductor memory device or device array of the like.

The above-described embodiments are for illustrative purposes, and those skilled in the art to which the above-described embodiments belong can easily transform into other specific forms without changing the technical idea or essential features of the above-described embodiments. You will understand. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope to be protected through this specification is indicated by the claims to be described later rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. should be interpreted as being

Claims (15)

1. an input/output unit that senses a user's motion; and

   From the detected movement of the user, a skeleton point, which is position information of body joints, is extracted and stored in a memory, and an eigen vector is calculated based on the skeleton point. A skeleton point-based service providing apparatus including a control unit that detects a change in movement posture.
2. According to claim 1,

   The control unit,

   Skeleton point-based service providing apparatus, characterized in that for determining whether the motion of the user is in an exercise state using a preset artificial intelligence technique (LSTM) based on the skeleton point.
3. According to claim 1,

   The control unit,

   Using the preset eigenvector, detecting a change in the user's movement posture in the movement motion of the user;

   An apparatus for providing a service based on skeleton points, which calculates calories according to a change in a user's movement posture based on the following equation.

   <mathematical expression>

   

   Here, result is the amount of calories consumed according to the change in the user's movement posture, O represents the amount of oxygen consumed, β is the amount of oxygen consumed for 1 minute by a body weight of 1 kg (MET), W is the mass, α is the gravitational acceleration, and t is time, g is the gravitational acceleration, w is the skeletal weight, and g' is the direction in which the joint moves relative to the direction of gravity.
4. According to claim 1,

   The control unit,

   A skeleton point-based service providing apparatus characterized in that an avatar is created based on the skeleton point, and the avatar is controlled to correspond to the user's motion.
5. According to claim 4,

   The control unit,

   The apparatus for providing a skeleton point-based service, characterized in that acquiring body information including any one of body shape and height based on the user's skeleton points, and generating an avatar corresponding to the acquired body information.
6. According to claim 5,

   The control unit,

   A skeleton point-based service providing apparatus that compares the generated avatar with a motion of a preset model and provides a service according to the comparison result.
7. A service providing method performed by a skeleton point based service providing apparatus,

   detecting a user's motion;

   extracting a skeleton point, which is position information of a body joint, from the detected motion of the user and storing it in a memory; and

   A service providing method comprising: calculating an eigen vector based on the skeleton points and detecting a change in the user's movement posture in the movement motion of the user.
8. According to claim 7,

   The step of detecting a change in the user's movement posture,

   and determining whether the motion of the user is in an exercise state by using a preset artificial intelligence method (LSTM) based on the skeleton point.
9. According to claim 7,

   The step of detecting a change in the user's movement posture,

   Based on the equation below, a service providing method characterized in that for calculating the calories according to the change in the movement posture of the user.

   <mathematical expression>

   

   Here, result is the amount of calories consumed according to the change in the user's movement posture, O represents the amount of oxygen consumed, β is the amount of oxygen consumed for 1 minute by a body weight of 1 kg (MET), W is the mass, α is the gravitational acceleration, and t is time, g is the gravitational acceleration, w is the skeletal weight, and g' is the direction in which the joint moves relative to the direction of gravity.
10. According to claim 7,

    The method,

    generating an avatar based on the skeleton point; and

    and controlling the avatar to correspond to the motion of the user.
11. According to claim 10,

    The step of creating the avatar,

    and obtaining body information including any one of a body shape and height based on the user's skeleton points, and generating an avatar corresponding to the acquired body information.
12. According to claim 11,

    The method,

    The service providing method further comprising the step of comparing the generated avatar with the operation of a preset model and providing a service according to the comparison result.
13. According to claim 7,

    The step of detecting a change in the user's movement posture,

    normalizing coordinates included in the skeleton points;

    collecting skeleton points including coordinates that have undergone the normalization process at regular time intervals, and acquiring eigenvectors at each joint using the collected skeleton points; and

    and detecting a change in the user's movement posture by smoothing the eigenvector obtained from each joint.
14. A computer readable recording medium in which a program for performing the method according to claim 7 is recorded.
15. A computer program stored in a recording medium to perform the method according to claim 7 and executed by a skeleton point-based service providing device.

Images