KR20220068282A - Intelligent virtual fitness system utilizing face recognition and motion recognition and operating method thereof - Google Patents

Abstract

The present invention relates to a technology capable of providing customized fitness coaching information to a user by analyzing an image input using a plurality of cameras installed in a fitness center. According to an embodiment of the present invention, a fitness server comprises: an information collection unit which collects image information related to a user using a plurality of cameras; an artificial intelligence processing unit which machine-learns the collected image information to recognize at least one of user identity information, movement information, motion information, and facial expression information, and based on the recognized at least one information, generates movement guidance information and coaching information; and a feedback information providing unit for providing the generated movement guidance information and the coaching information to a user terminal. The present invention can improve user convenience by checking member information through face recognition.

Description

Intelligent virtual fitness system using face recognition and motion recognition and its operation method

The present invention relates to an intelligent virtual fitness system using face recognition and motion recognition, and more specifically, by analyzing input images using a plurality of cameras installed in a fitness center, it is possible to provide fitness coaching information customized to the user. It's about technology.

Recently, systems that provide intelligent services by utilizing image information using cameras have emerged.

In particular, the motion recognition technology is being developed as a technology suitable for commercialization as the motion recognition rate increases with the development of deep learning algorithms.

As interest in health has grown, many people are aware of the need for exercise for health promotion, and are actually managing their health through various exercises.

It is common to use a fitness center or a health club for body management, and in the fitness center, you can do systematic exercise with a trainer using various exercise equipment.

In particular, when exercising with a trainer who has professional knowledge to create a healthy body, the popularity of personal training continues to increase because the effect of exercise appears quickly.

Conventionally, in a form in which trainers in a fitness center directly instruct a user who wants exercise guidance to exercise through face-to-face, there is a problem that trainers must be directly hired in the fitness center, and costs may occur greatly from the user's point of view.

In addition, in relation to the recent corona crisis, when a user wants to exercise alone without face-to-face with other people, there is a hassle of counting or correcting by looking in a mirror for the exercise posture, number of times, etc.

In addition, since it is difficult to automatically record one's own exercise and process it in a big data format, there is considerable variation among trainers even in recommending one's own customized exercise.

In order to increase the effect of exercise, it is necessary to create an exercise schedule, set an exercise goal, and provide appropriate feedback by referring to the amount of exercise or physical condition.

However, in the related art, even when receiving personal training, there is a problem in that it is not efficient because it is impossible to obtain appropriate feedback information about one's own exercise amount and exercise effect, and scientific and systematic exercise is impossible.

Korean Patent Publication No. 10-2019-0029430, "AI based online fitness system and personal training method" Korean Patent Registration No. 10-1931724, "Health Navigation System" Korea Patent No. 10-1970687, "Fitness coaching system using personalized augmented reality technology" Korean Patent No. 10-2117708, "Personal Exercise Management System"

An object of the present invention is to provide an intelligent virtual fitness system using face recognition and motion recognition and an operation method thereof.

An object of the present invention is to improve user convenience by checking member information through face recognition.

The present invention not only analyzes the user's motion using a plurality of cameras installed on the ceiling or the upper part of the wall of the fitness center and the camera installed on the exercise equipment, but also analyzes the movement route, so that the utilization related to the share of the exercise equipment in the fitness center aims to improve

An object of the present invention is to improve the exercise efficiency of a user by recognizing a user's facial expression using a camera installed in an exercise equipment, and providing customized encouragement information according to the recognized facial expression.

An object of the present invention is to improve exercise efficiency by recognizing a user's motion using a plurality of cameras and providing customized coaching information according to the recognized motion.

An object of the present invention is to automatically store the user's exercise result information and provide recommendation information for recommending a customized exercise according to the formation of data in the form of big data.

The present invention provides customized coaching information in the future by storing the user's exercise time, number of exercise, exercise type and routine, etc., and quantifying the degree of difficulty through data linkage with facial recognition in order to perform artificial intelligence-based big data analysis. aim to

An object of the present invention is to provide customized coaching information to a user using artificial intelligence without using a personal trainer in a fitness center.

An object of the present invention is to provide customized coaching information at a low cost in a fitness center.

A fitness server using face recognition and motion recognition according to an embodiment of the present invention includes an information collecting unit that collects image information related to a user using a plurality of cameras, machine learning the collected image information to identify the user Information, movement route information, motion information, and facial expression information recognizes at least one of the information, based on the recognized at least one information, the artificial intelligence processing unit and the user terminal for generating the movement guide information and coaching information, the generated movement guide It may include a feedback information providing unit that provides information and coaching information.

The information collection unit collects image information related to at least one of the user's face, location, and motion through at least one camera installed on a ceiling or an upper portion of a wall of a fitness center among the plurality of cameras, and the plurality of cameras Image information related to at least one of the user's facial expression and motion may be collected through a camera installed in each of a plurality of exercise equipment in the fitness center.

The artificial intelligence processing unit recognizes the user's face by machine learning image information related to at least one of the user's face, location, and motion to determine the identity information, and then determines whether the user is a registered customer, and the user's location After determining the movement information determined by recognizing a change, determining an exercise equipment in an vacant state among a plurality of exercise equipment to generate the movement guide information, and recognizing the change in the user's movement and the coaching related to the movement information information can be created.

The artificial intelligence processing unit may give an identification number to the user determined as the registered customer, and generate the movement guide information and the coaching information in consideration of the assigned identification number.

The artificial intelligence processing unit may detect the face at a head position based on the machine-learned image information, extract the detected embedding of the face, and then perform face recognition.

The artificial intelligence processing unit derives the sum of weights by machine learning sensing information of sensors installed in each of the plurality of exercise equipment and image information on the plurality of exercise equipments collected through the plurality of cameras, and the derivation It is possible to determine the exercise equipment in the vacant state based on the sum of the weights.

When the user is determined as the registered customer, the artificial intelligence processing unit may update the movement guide information based on at least one of the user's exercise history and a pre-planned exercise routine for the user. .

The artificial intelligence processing unit generates a heat map for each joint candidate related to the user's motion by machine learning image information related to at least one of the user's facial expression and motion, and connects to each joint candidate. It is possible to determine the joint coordinates according to securing the , and determine the motion information according to the change in the determined joint coordinates.

The artificial intelligence processing unit may count the number of exercises or determine a posture mismatch by machine learning the determined motion information and predetermined motion information for each of the plurality of exercise equipment.

The artificial intelligence processing unit may generate coaching information related to posture correction when it is determined as the posture mismatch.

The artificial intelligence processing unit may recognize a change in the user's facial expression based on the facial expression information, and generate encouragement information according to the recognized facial expression change.

According to an embodiment of the present invention, an intelligent virtual fitness system using face recognition and motion recognition collects image information related to a user using a plurality of cameras, and machine-learns the collected image information to identify the user , recognizes at least one of information among movement information, motion information, and facial expression information, generates movement guide information and coaching information based on the recognized at least one information, and generates movement guide information and coaching information with a user terminal It may include a user terminal that is connected to the fitness server through a fitness server and a coaching application providing

The fitness server collects image information related to at least one of the user's face, location, and motion through at least one camera installed on a ceiling or an upper portion of a wall of a fitness center among the plurality of cameras, and the plurality of cameras Image information related to at least one of the user's facial expression and motion may be collected through a camera installed in each of a plurality of exercise equipment in the fitness center.

The fitness server recognizes the user's face by machine learning image information related to at least one of the user's face, location, and motion to determine the identity information, then determines whether the user is a registered customer, and changes the user's location After determining the movement route information determined by recognizing the can create

The fitness server derives the sum of weights by machine learning sensing information of sensors installed in each of the plurality of exercise equipment and image information about the plurality of exercise equipment collected through the plurality of cameras, and the derived Determine the exercise equipment in the vacant state based on the sum of the weights, when the user is determined as the registered customer, at least one of the user's exercise history and a pre-planned exercise routine for the user Based on it, the movement guide information may be updated.

The fitness server generates a heat map for each joint candidate related to the user's motion by machine learning image information related to at least one of the user's facial expression and motion, and provides connectivity to each joint candidate. Determine the joint coordinates according to the securing, and determine the motion information according to the change in the determined joint coordinates, counting the number of motions by machine learning the determined motion information and predetermined motion information for each of the plurality of exercise devices, or A posture mismatch may be determined, and when the posture mismatch is determined, coaching information related to posture correction may be generated.

The user terminal may use a text to speech (TTS) function of the coaching application to provide the generated movement route guide information and the coaching information to the user by voice.

According to an embodiment of the present invention, a method of operating a fitness server includes: in the operating method of a fitness server using face recognition and motion recognition, collecting, in an information collecting unit, image information related to a user by using a plurality of cameras , the artificial intelligence processing unit recognizes at least one of the user's identity information, movement route information, motion information, and facial expression information by machine learning the collected image information, and guides the movement route based on the recognized at least one information It may include the step of generating information and coaching information and providing the generated movement guide information and coaching information to the user terminal in the feedback information providing unit.

The collecting of image information related to the user by using the plurality of cameras may include at least one of the user's face, location, and motion through at least one camera installed on a ceiling or an upper part of a wall of a fitness center among the plurality of cameras. It may include collecting image information related to and collecting image information related to at least one of the user's facial expression and motion through a camera installed in each of a plurality of exercise equipment in the fitness center among the plurality of cameras. .

Machine learning the collected image information to recognize at least one of the user's identity information, movement route information, motion information, and facial expression information, and generate movement route guide information and coaching information based on the recognized at least one information The step of generating a heat map for each joint candidate related to the user's motion by machine learning image information related to at least one of the user's facial expression and motion, Determine the joint coordinates according to the securing, and determine the motion information according to the change in the determined joint coordinates, counting the number of motions by machine learning the determined motion information and predetermined motion information for each of the plurality of exercise devices, or The method may include determining the posture mismatch and generating coaching information related to posture correction when the posture mismatch is determined.

The present invention can provide an intelligent virtual fitness system using face recognition and motion recognition and an operation method thereof.

The present invention can improve user convenience by checking member information through face recognition.

The present invention not only analyzes the user's motion using a plurality of cameras installed on the ceiling or the upper part of the wall of the fitness center and the camera installed on the exercise equipment, but also analyzes the movement route, so that the utilization related to the share of the exercise equipment in the fitness center can improve

The present invention can improve the exercise efficiency of the user by recognizing the user's facial expression using a camera installed in the exercise equipment, and providing customized encouragement information according to the recognized facial expression.

The present invention can improve exercise efficiency by recognizing a user's motion using a plurality of cameras and providing customized coaching information according to the recognized motion.

According to the present invention, the user's exercise result information is automatically stored to form big data type data, so that it is possible to provide recommendation information for recommending a customized exercise.

The present invention stores the user's exercise time, exercise frequency, exercise type and routine, etc. for artificial intelligence-based big data analysis, and quantifies the degree of difficulty through data linkage with expression recognition to provide customized coaching information in the future. can

The present invention can provide customized coaching information to a user using artificial intelligence without using a personal trainer in a fitness center.

The present invention can provide customized coaching information at a low cost in a fitness center.

1 and 2 are diagrams illustrating an intelligent virtual fitness system according to an embodiment of the present invention.
3 is a diagram illustrating a fitness server according to an embodiment of the present invention.
4 to 7 are diagrams for explaining a method of operating a fitness server according to an embodiment of the present invention.
8 is a view for explaining a method of operating an intelligent virtual fitness system according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed herein are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiment according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Expressions describing the relationship between elements, for example, “between” and “between” or “directly adjacent to”, etc. should be interpreted similarly.

The terminology used herein is used only to describe specific embodiments, and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, stage, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of stages, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference numerals in each figure indicate like elements.

1 and 2 are diagrams illustrating an intelligent virtual fitness system according to an embodiment of the present invention.

1 illustrates a schematic diagram of an intelligent virtual fitness system according to an embodiment of the present invention;

Referring to FIG. 1 , the intelligent virtual fitness system 100 according to an embodiment of the present invention includes a fitness server 110 , a first camera 120 , a second camera 122 , a first exercise device 130 , It includes a second exercise device 132 , a third exercise device 134 , and a user terminal 140 .

For example, each of the first exercise device 130 , the second exercise device 132 , and the third exercise device 134 includes a camera 131 , a camera 133 , and a camera 135 .

For example, the second exercise device 132 and the third exercise device 134 may be identified as empty exercise equipment based on sensor information and image information based on the first camera 120 and the second camera 122 . have.

According to an embodiment of the present invention, the first camera 120 and the second camera 122 may be installed on the ceiling or the wall of the fitness center.

The first camera 120 and the second camera 122 collect image information for checking the user's movement and empty exercise equipment in the fitness center, and provide the collected image information to the fitness server 110 through the network. can do.

Also, when a user enters the fitness center, the first camera 120 and the second camera 122 collect image information for recognizing the user's face.

Accordingly, the fitness server 110 may identify the user's identity by recognizing the user's face based on the collected image information.

In addition, the image information collected by the first camera 120 and the second camera 122 may be utilized for security in the fitness center.

For example, when the fitness server 110 fails to verify the identity of a person entering the fitness center, the fitness server 110 may generate alarm information to maintain security in the fitness center.

For example, the camera 131 , the camera 133 , and the camera 135 collect image information about the user when the user uses the exercise equipment.

The image information based on the camera 131 , the camera 133 , and the camera 135 may be utilized to recognize the user's motion by estimating the joint position, and to check the user's facial expression.

That is, the fitness server 110 recognizes the user's motion finely and accurately by machine learning the image information based on the camera 131, the camera 133, and the camera 135, so that the wrong pose when exercising for each exercise equipment is detected. If there is, it can inform the user and induce the user to exercise with the correct posture.

For example, the camera 131 , the camera 133 , and the camera 135 may collect image information about the user located in front when the user starts to use the exercise equipment.

According to an embodiment of the present invention, the fitness server 110 collects the first camera 120 and the second camera 122 and the camera 131, the camera 133 and the camera 135 installed in each exercise device. It is possible to provide a virtual training service by generating movement guide information and coaching information by machine learning the obtained image information.

According to an embodiment of the present invention, the fitness server 110 is linked with the user terminal 140 to allow the user to use an exercise device that artificial intelligence of the fitness server 110 informs according to an existing exercise history and an already planned exercise routine. It is possible to provide movement guide information to move and exercise.

Here, the fitness server 110 may first guide the empty exercise equipment using the image information input from the first camera 120 and the second camera 122 .

The fitness server 110 not only determines whether another user is using or occupies the exercise equipment using camera image information, but also attaches a sensor to the exercise equipment itself at each person's seat to determine whether the exercise equipment is occupied or not. can do.

Here, the fitness server 110 determines the value input from the sensor and the value input to the image in an artificial intelligence algorithm by appropriately using a weighted sum, and the value of this weight may be updated through learning.

In addition, the fitness server 110, when the user exercises on the exercise equipment, the artificial intelligence learns in advance the movements suitable for each exercise equipment and stores the data as data. You can provide feedback information.

In addition, the fitness server 110 guides the user to another exercise equipment based on the artificial intelligence of the fitness server 110 when all exercise sets are finished by the user. In this case, the user may first guide the user to an empty exercise equipment.

In addition, the fitness server 110 may recognize the facial expression of the user when exercising for every exercise equipment, adjust the voice message to the user so that an appropriate operation can be performed when a difficult expression is seen.

For example, the camera 131 , the camera 133 , the camera 135 , the first camera 120 , and the second camera 122 may be at least one of a 2D camera and a 3D camera.

According to an embodiment of the present invention, the fitness server 110 extracts the user's skeleton and joint information using a plurality of cameras for motion recognition of multiple users, and based on the extracted information, through a deep learning neural network According to machine learning, it is possible to finally determine whether the user performs the motion by identifying a motion suitable for each exercise device.

For example, the fitness server 110 may reduce the number of cameras and recognize the motion even in the case of the back view by dividing the back view and the front view when drawing human skeleton information in order to recognize the motion of a plurality of users.

The fitness server 110 stores the user's exercise time, the number of exercises, the exercise type and routine, etc. for big data analysis, and quantifies the degree of difficulty through data linkage with facial recognition recognition to apply customized services in the future. .

In addition, the fitness server 110 may precisely analyze the user's motion to coach whether the motion in each exercise device is accurately performed, and if the motion is incorrect, may provide coaching information for coaching.

According to an embodiment of the present invention, the user terminal 140 is operated based on a coaching application, which is connected to the fitness server 110 in the fitness center through a network, and provides AI-based coaching to users. support you to do it

In addition, the identification number of the user terminal 140 is interlocked with the user's face recognition to support skipping the authentication process in the coaching application.

2 illustrates a block diagram of an intelligent virtual fitness system according to an embodiment of the present invention.

Referring to FIG. 2 , the intelligent virtual fitness system 200 according to an embodiment of the present invention includes a fitness server 210 and a user terminal 220 .

As an example, the intelligent virtual fitness system 200 performs face recognition and motion recognition by machine learning images obtained through a plurality of cameras, and the fitness server 210 generates customized information based on face recognition and motion recognition. and provides the generated customized information to the user through the user terminal 220 .

According to an embodiment of the present invention, the fitness server 210 collects image information related to a user using a plurality of cameras, and analyzes the collected image information to identify the user's identity information, movement information, motion information, and facial expression information. at least one piece of information is recognized.

In addition, the fitness server 210 may generate movement guide information and coaching information based on the recognized at least one piece of information, and may provide the generated movement guide information and coaching information to the user terminal 220 .

As an example, the fitness server 210 may perform machine learning by collecting image information related to at least one of the user's face, location, and motion through at least one camera installed on the ceiling or wall of the fitness center among a plurality of cameras. can

In addition, the fitness server 210 may collect image information related to at least one of a user's facial expression and motion through a camera installed on each of a plurality of exercise equipment in the fitness center among the plurality of cameras.

According to an embodiment of the present invention, the fitness server 210 recognizes the user's face by machine learning image information related to at least one of the user's face, location, and motion, identifies the identity information, and then determines whether the user is a registered customer. can

In addition, the fitness server 210 may determine the movement line information determined by recognizing the change in the user's location, and then determine the exercise equipment in the vacant state among a plurality of exercise equipment to generate the movement guide information.

Also, the fitness server 210 may generate the coaching information related to the motion information by recognizing a change in the user's motion.

According to an embodiment of the present invention, the fitness server 210 machine-learns image information about a plurality of exercise equipments collected through a plurality of cameras and sensing information of a sensor installed in each of a plurality of exercise equipments, and sums the weights. , and determine the exercise equipment in the vacant state based on the sum of the derived weights, but update the movement guide information based on at least one of the user's exercise history and the user's pre-planned exercise routine for the user can do.

As an example, the fitness server 210 generates a heat map for each joint candidate related to the user's motion when the user performs an exercise using cameras respectively installed in a plurality of exercise equipment, and calculates joint coordinates according to the heat map. By determining, it is possible to machine learn motion information according to changes in joint coordinates, count the number of exercises, or determine a posture mismatch to generate and provide coaching information related to posture correction.

According to an embodiment of the present invention, the user terminal 220 is connected to the fitness server 210 through a coaching application, and provides the movement guide information and the upper coaching information generated by the fitness server 210 as voice information to the user. can do.

For example, the user terminal 220 may use a text to speech (TTS) function of the coaching application to provide movement guide information and coaching information to the user by voice.

3 is a diagram illustrating a fitness server according to an embodiment of the present invention.

3 illustrates the components of a fitness server according to an embodiment of the present invention.

Referring to FIG. 3 , the fitness server 300 according to an embodiment of the present invention includes an information collecting unit 310 , an artificial intelligence processing unit 320 , a feedback providing unit 330 , and a storage unit 340 .

According to an embodiment of the present invention, the information collection unit 310 collects image information related to a user by using a plurality of cameras.

For example, the plurality of cameras may include at least one camera installed on a ceiling or an upper portion of a wall of the fitness center and a camera installed on each of a plurality of exercise equipment in the fitness center.

For example, the information collection unit 310 may collect image information related to at least one of a user's face, location, and motion through at least one camera installed on a ceiling or an upper portion of a wall of a fitness center among a plurality of cameras.

The information collection unit 310 according to an embodiment of the present invention may collect image information related to at least one of a user's facial expression and motion through a camera installed on each of a plurality of exercise equipment in a fitness center among a plurality of cameras.

The artificial intelligence processing unit 320 according to an embodiment of the present invention recognizes at least one of the user's identity information, movement line information, motion information, and facial expression information by machine learning the collected image information, and the recognized at least Based on one piece of information, it is possible to generate movement guide information and coaching information.

For example, the artificial intelligence processing unit 320 may recognize the user's face by machine learning image information related to at least one of the user's face, location, and motion to determine identity information, and then determine whether the user is a registered customer.

In addition, the artificial intelligence processing unit 320 determines the movement information determined by recognizing the change in the user's position, and then determines the exercise equipment in the vacant state among a plurality of exercise equipment to generate movement information, and to change the user's movement. By recognizing it, it is possible to generate coaching information related to the motion information.

According to an embodiment of the present invention, the artificial intelligence processing unit 320 generates a heat map for each joint candidate related to the user's motion by machine learning image information related to at least one of the user's facial expression and motion, and , may determine joint coordinates according to securing connectivity for each joint candidate, and may determine motion information according to a change in the determined joint coordinates.

In addition, the artificial intelligence processing unit 320 may machine-learning the determined motion information and the predetermined motion information for each of the plurality of exercise equipment to count the number of exercises or determine the posture mismatch.

For example, the artificial intelligence processing unit 320 counts the number of workouts when the motion information determined based on the image and the reference information determined by machine learning the predetermined motion information match.

In addition, when the reference information determined by machine learning the motion information determined based on the image and the motion information determined in advance, the artificial intelligence processing unit 320 identifies the mismatched area, and customized coaching information related to the mismatched area to create

Accordingly, when the artificial intelligence processing unit 320 is determined to be a posture mismatch, it is possible to generate coaching information related to posture correction.

According to an embodiment of the present invention, the artificial intelligence processing unit 320 may give an identification number to a user determined as a registered customer, and generate movement guide information and coaching information in consideration of the assigned identification number.

That is, the artificial intelligence processing unit 320 may generate movement guide information and coaching information to manage movement and coaching of the user in the fitness center based on the identification number assigned to each user.

According to an embodiment of the present invention, the artificial intelligence processing unit 320 identifies the user's head position based on the machine-learned image information, detects a face at the identified head position, and extracts the detected face embedding to make the face recognition can be performed.

For example, the artificial intelligence processing unit 320 may generate identification information about the user by using the extracted face.

According to an embodiment of the present invention, the artificial intelligence processing unit 320 performs machine learning on sensing information of a sensor installed in each of a plurality of exercise equipment and image information on a plurality of exercise equipments collected through the plurality of cameras to perform weighting. You can derive the sum of and determine the exercise equipment in the vacant state based on the sum of the derived weights.

For example, the artificial intelligence processing unit 320 may update the sum of the weights through machine learning.

As an example, when the user is determined as a registered customer, the artificial intelligence processing unit 320 may update the movement guide information based on at least one of a user's exercise history and a pre-planned exercise routine for the user. .

For example, the artificial intelligence processing unit 320 guides the movement to guide the user to the second exercise equipment to be used in the next turn if the exercise equipment is not vacant, even if the exercise is performed using the first exercise equipment in the user's exercise record and exercise routine. Information can be updated.

According to an embodiment of the present invention, the artificial intelligence processing unit 320 may recognize a change in the user's facial expression based on the facial expression information, and generate encouragement information according to the recognized facial expression change.

According to an embodiment of the present invention, the feedback providing unit 330 may provide the user terminal with movement guide information and coaching information.

For example, the feedback providing unit 330 may serve as a communication unit, and may be connected to a user terminal through a communication network such as Bluetooth or Wi-Fi, and may provide customized movement guide information and coaching information for each connected user terminal.

According to an embodiment of the present invention, the storage unit 340 may store all information data related to the machine-learned data in the artificial intelligence processing unit 320 .

For example, the storage unit 340 may store information related to a user's exercise record, exercise routine, exercise time, and the like.

Also, the storage unit 340 stores an identification number corresponding to the user generated after face recognition, and when the user later visits the fitness center, provides the stored identification number to improve the user recognition rate.

For example, the storage unit 340 may also be referred to as a big data platform.

Therefore, the present invention not only analyzes the user's motion using a plurality of cameras installed on the ceiling or the upper part of the wall of the fitness center and the camera installed on the exercise equipment, but also analyzes the movement, thereby increasing the share of exercise equipment in the fitness center and related usability can be improved.

In addition, the present invention can improve the exercise efficiency of the user by recognizing the user's facial expression using a camera installed in the exercise equipment, and providing customized encouragement information according to the recognized facial expression.

In addition, the present invention can improve exercise efficiency by recognizing a user's motion using a plurality of cameras and providing customized coaching information according to the recognized motion.

In addition, the present invention can provide recommendation information for recommending a customized exercise by automatically storing the user's exercise result information to form big data type data.

In addition, the present invention can improve user convenience by checking member information through face recognition.

4 to 7 are diagrams for explaining a method of operating a fitness server according to an embodiment of the present invention.

4 is a fitness server according to an embodiment of the present invention, after collecting image information, machine learning the collected image information to generate movement guide information and coaching information, and then feedback the generated movement guide information and coaching information exemplifies the action.

Referring to FIG. 4 , in the operation method of the fitness server, image information is collected in step 401 .

That is, the operating method of the fitness server collects image information related to a user by using a plurality of cameras.

The method of operation of the fitness server in step 402 generates movement guide information and coaching information.

That is, the operation method of the fitness server recognizes at least one of the user's identity information, movement route information, motion information, and facial expression information by machine learning the image information collected in step 401 to generate movement guide information and coaching information. can do.

In the operation method of the fitness server in step 403 , the movement guide information and coaching information generated in step 402 may be provided to the user terminal.

For example, the movement guide information may include path information on which the user moves in the fitness center.

In addition, the coaching information may include correction information for correcting a user's wrong posture, encouragement information for encouraging the user's exercise state, and guide information for guiding the user's use of an exercise equipment.

5 illustrates an embodiment in which a method of operating a fitness server according to an embodiment of the present invention provides movement guide information.

Referring to FIG. 5 , in the method of operating a fitness server according to an embodiment of the present invention, image information is collected in step 501 .

That is, the operating method of the fitness server collects image information related to a user by using a plurality of cameras.

In step 502, the method of operating the fitness server detects face information and exercise equipment information from image information.

That is, the operating method of the fitness server detects a face region from the user's image information by machine learning image information, and detects face information for recognizing a face from the detected face region.

In addition, the operating method of the fitness server detects the exercise equipment information to identify the empty exercise equipment in the fitness center by machine learning the image information.

In step 503, the operating method of the fitness server determines whether face information is recognized.

That is, the operating method of the fitness server determines whether or not facial information sufficient to confirm the user's identity is recognized using the face information detected in step 502 and at the same time determines whether the user's identity is confirmed. .

Here, the method of operating the fitness server proceeds to step 504 when face information is recognized, and returns to step 502 to detect face information again when face information is not recognized.

In the operation method of the fitness server in step 504, it is determined whether there is an empty exercise equipment.

That is, the operating method of the fitness server determines the existence of an empty exercise equipment based on the exercise equipment information detected in step 502, and if there is an empty exercise equipment, proceeds to step 505, and the empty exercise equipment If there is no, it returns to step 502 to detect the exercise equipment information again.

In the operation method of the fitness server in step 505, an identification number is given to a face-recognized user, and guide information related to an empty exercise equipment is provided.

That is, the operation method of the fitness server determines the face-recognized user as a member, and comprehensively determines the schedule information based on the member information and the information of the empty exercise equipment to provide the user with the movement guide information that is the guide information related to the empty exercise equipment. provided to the user through the terminal.

6 illustrates an embodiment in which a method of operating a fitness server according to an embodiment of the present invention provides coaching information.

Referring to FIG. 6 , in the method of operating a fitness server according to an embodiment of the present invention, image information is collected in step 601 .

That is, the operating method of the fitness server collects image information related to a user by using a plurality of cameras.

In the operation method of the fitness server in step 602, a heat map is generated by machine learning image information, and a user's motion is recognized using joint coordinates on the heat map.

That is, the operation method of the fitness server machine-learns the image information collected through a camera attached to each exercise device. In the machine learning process, a heat map is generated based on information related to the body joint information model, and the collected image information After extracting each joint candidate from , the user's motion is recognized by recognizing a gesture using the extracted joint candidate with joint coordinates and a heat map.

In step 603, the method of operating the fitness server determines whether the recognized motion matches the reference motion.

That is, in the operation method of the fitness server, it is determined whether the user's motion recognized through machine learning in step 602 and a reference motion corresponding to an exercise motion suitable for each exercise equipment match, and if they match, step 605 , and if there is a discrepancy, the process proceeds to step 604 .

In step 604, the method of operation of the fitness server provides coaching information related to posture correction.

That is, in the method of operating the fitness server, after providing feedback information related to posture correction to the user through voice or text, the method returns to step 603 and checks whether the feedback information is reflected.

In step 605, the method of operation of the fitness server counts the number of exercises.

That is, in the operating method of the fitness server, when the current user performs an exercise suitable for the exercise method of the exercise equipment, counting is performed.

The method of operation of the fitness server in step 606 provides exercise result information.

That is, in the operating method of the fitness server, the exercise effect related information obtained by the user through the current exercise equipment together with the counting information determined in step 605 is provided as exercise result information.

7 illustrates an embodiment in which a method of operating a fitness server according to an embodiment of the present invention provides encouragement information.

Referring to FIG. 7 , in the method of operating a fitness server according to an embodiment of the present invention, image information is collected in step 701 .

That is, the operating method of the fitness server collects image information related to the user by using a camera installed in the exercise equipment.

In step 702, the operation method of the fitness server recognizes the user's facial expression information by machine learning the image information.

That is, the operation method of the fitness server recognizes a change in the user's facial expression in the image information by machine learning the image information continuously collected through the camera installed in the exercise equipment.

In step 703, the method of operation of the fitness server determines whether the expression changes.

That is, the method of operation of the fitness server proceeds to step 704 when a change in the user's facial expression occurs, and omits step 704 when there is no facial expression change.

The method of operation of the fitness server in step 704 provides encouragement information.

That is, the operating method of the fitness server determines that the change in the user's facial expression is due to the intensity of the exercise, and provides an encouraging comment for improving the user's exercise power as encouragement information.

For example, the encouragement information may include "Cheer up", "You're done", "Fighting", "You're cool", and the like.

The method of operation of the fitness server in step 705 provides exercise result information.

That is, the operating method of the fitness server provides exercise effect related information obtained by the user through the current exercise equipment as exercise result information.

For example, the exercise effect related information may include a calorie consumption rate, a body fat reduction rate, a muscle mass increase rate, and the like.

8 is a view for explaining a method of operating an intelligent virtual fitness system according to an embodiment of the present invention.

8 illustrates a method of operating an intelligent virtual fitness system according to an embodiment of the present invention.

Referring to FIG. 8 , in step S801 , the fitness server 800 transmits coaching application installation information to the user terminal 820 side based on user recognition.

That is, the fitness server 800 transmits installation information for inducing installation of the coaching application so that the user terminal 820 device can interwork with the fitness server 800 .

In step S802, the user terminal 820 is connected to the fitness server through the installed coaching application.

In step (S803), the fitness server 800 transmits exercise equipment guide information to the user terminal 820 .

Here, the fitness server 800 transmits the exercise equipment guide information to the user terminal 820 by machine learning sensing information and image information for a plurality of exercise equipment in the fitness center.

In step 804 , the exercise information collecting device 810 collects the user's image information, and transmits the collected image information to the fitness server 800 .

For example, the exercise information collection device 810 may include a sensor and a camera disposed on each of a plurality of exercise equipment, and a plurality of cameras disposed on a ceiling in a fitness center.

In step 805 , the fitness server 800 transmits feedback information including coaching information and encouragement information to the user terminal 820 .

For example, when determining the posture mismatch based on the information collected by the exercise information collecting device 810 , the fitness server 800 generates coaching information related to the posture mismatch and transmits the generated coaching information to the user terminal 820 .

In addition, when the fitness server 800 detects a change in the user's facial expression that distorts the user's facial expression, the user may transmit encouragement information related to a comment saying "You are doing well" or a comment "Stay a little stronger" to the user terminal 820 . .

In step 806, the exercise information collecting device 810 collects image information related to the feedback information, and transmits the collected image information again.

In step 807 , the fitness server 800 checks whether the feedback information is reflected, and then transmits the exercise result information and guide information related to the exercise schedule to the user terminal 820 .

For example, guide information related to an exercise schedule may correspond to exercise-related schedule information to be performed after an exercise performed by the user.

That is, the fitness server 800 provides exercise result information to the user while also detecting a change in the user's state according to the feedback information.

Accordingly, the present invention can provide an intelligent virtual fitness system using face recognition and motion recognition and an operation method thereof.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

200: fitness system 210: fitness server
220: user terminal

Claims (20)

In a fitness server using face recognition and motion recognition,
an information collection unit for collecting image information related to a user by using a plurality of cameras;
Machine learning the collected image information to recognize at least one of the user's identity information, movement route information, motion information, and facial expression information, and generate movement route guide information and coaching information based on the recognized at least one information artificial intelligence processing unit; and
Comprising a feedback information providing unit for providing the generated movement guide information and coaching information to a user terminal
fitness server. According to claim 1,
The information collection unit
Collect image information related to at least one of the user's face, location, and motion through at least one camera installed on a ceiling or an upper portion of a wall of the fitness center among the plurality of cameras, and a plurality of cameras in the fitness center among the plurality of cameras Characterized in collecting image information related to at least one of the user's facial expression and motion through a camera installed in each of the exercise equipment of
fitness server. 3. The method of claim 2,
The artificial intelligence processing unit
After recognizing the user's face by machine learning image information related to at least one of the user's face, location, and motion to determine the identity information, determining whether the user is a registered customer, and recognizing a change in the user's location After determining the movement information, determining an exercise equipment in a vacant state among a plurality of exercise equipment to generate the movement guide information, recognizing the change in the user's movement and generating the coaching information related to the movement information characterized
fitness server. 4. The method of claim 3,
The artificial intelligence processing unit
Assigning an identification number to the user determined as the registered customer, and generating the movement guide information and the coaching information in consideration of the assigned identification number
fitness server. 4. The method of claim 3,
The artificial intelligence processing unit
Detecting the face at a head position based on the machine-learned image information, extracting the detected face embedding, and performing face recognition
fitness server. 4. The method of claim 3,
The artificial intelligence processing unit
The sum of weights is derived by machine learning the sensing information of the sensors installed in each of the plurality of exercise equipments and the image information about the plurality of exercise equipments collected through the plurality of cameras, and the sum of the derived weights is Characterized in determining the exercise equipment of the vacant state based on
fitness server. 4. The method of claim 3,
The artificial intelligence processing unit
When the user is determined as the registered customer, the movement guide information is updated based on at least one of the user's exercise history and a pre-planned exercise routine for the user.
fitness server. 3. The method of claim 2,
The artificial intelligence processing unit
Joints according to machine learning image information related to at least one of the user's facial expressions and motions to generate a heat map for each joint candidate related to the user's motion, and securing connectivity for each joint candidate Coordinates are determined, and the motion information according to the determined change in joint coordinates is determined.
fitness server. 9. The method of claim 8,
The artificial intelligence processing unit
Counting the number of exercises or determining posture mismatch by machine learning the determined motion information and predetermined motion information for each of the plurality of exercise equipment
fitness server. 10. The method of claim 9,
The artificial intelligence processing unit
When it is determined as the posture mismatch, coaching information related to posture correction is generated.
fitness server. 3. The method of claim 2,
The artificial intelligence processing unit
Recognizing a change in the user's facial expression based on the facial expression information, and generating encouragement information according to the recognized facial expression change
fitness server. In an intelligent virtual fitness system using face recognition and motion recognition,
Collecting image information related to a user using a plurality of cameras, analyzing the collected image information to recognize at least one of the user's identity information, movement line information, motion information, and facial expression information, and a fitness server that generates movement guide information and coaching information based on one piece of information, and provides the generated movement guide information and coaching information to a user terminal; and
Connected to the fitness server through a coaching application, characterized in that it comprises a user terminal for providing the generated movement guide information and the coaching information to the user
fitness system. 13. The method of claim 12,
the fitness server
Collect image information related to at least one of the user's face, location, and motion through at least one camera installed on a ceiling or an upper portion of a wall of the fitness center among the plurality of cameras, and a plurality of cameras in the fitness center among the plurality of cameras Characterized in collecting image information related to at least one of the user's facial expression and motion through a camera installed in each of the exercise equipment of
fitness system. 14. The method of claim 13,
the fitness server
After recognizing the user's face by machine learning image information related to at least one of the user's face, location, and motion to determine the identity information, determining whether the user is a registered customer, and recognizing a change in the user's location After determining the movement information, determine the exercise equipment in a vacant state among a plurality of exercise equipment to generate the movement guide information, and to recognize the change in the user's movement to generate the coaching information related to the movement information characterized
fitness system. 15. The method of claim 14,
the fitness server
The sum of weights is derived by machine learning the sensing information of the sensors installed in each of the plurality of exercise equipments and the image information about the plurality of exercise equipments collected through the plurality of cameras, and the sum of the derived weights is Determine the exercise equipment in the vacant state based on, but when the user is determined as the registered customer, the movement line based on at least one of the user's exercise history and a pre-planned exercise routine for the user Updating guide information
fitness system. 13. The method of claim 12,
the fitness server
Joints according to machine learning image information related to at least one of the user's facial expressions and motions to generate a heat map for each joint candidate related to the user's motion, and securing connectivity for each joint candidate Determine the coordinates, and determine the motion information according to the change in the determined joint coordinates, by machine learning the determined motion information and predetermined motion information for each of the plurality of exercise equipment, counting the number of movements or determining a posture mismatch and generating coaching information related to posture correction when it is determined as the posture mismatch
fitness system. 13. The method of claim 12,
The user terminal is
Using the TTS (Text To Speech) function of the coaching application, the generated movement guide information and the coaching information are provided to the user by voice
fitness system. In the operation method of a fitness server using face recognition and motion recognition,
collecting, in the information collecting unit, image information related to a user by using a plurality of cameras;
The artificial intelligence processing unit analyzes the collected image information to recognize at least one of the user's identity information, movement information, motion information, and facial expression information, and based on the recognized at least one information, movement guide information and generating coaching information; and
In the feedback information providing unit, comprising the step of providing the generated movement guide information and coaching information to the user terminal
How a fitness server works. 19. The method of claim 18,
The step of collecting image information related to a user by using the plurality of cameras includes:
collecting image information related to at least one of the user's face, location, and motion through at least one camera installed on a ceiling or an upper portion of a wall of a fitness center among the plurality of cameras; and
Comprising the step of collecting image information related to at least one of the user's facial expression and motion through a camera installed in each of a plurality of exercise equipment in the fitness center among the plurality of cameras
How a fitness server works. 19. The method of claim 18,
Analyzing the collected image information to recognize at least one of the user's identity information, movement route information, motion information, and facial expression information, and generate movement route guide information and coaching information based on the recognized at least one information step is
generating a heat map for each joint candidate related to the user's motion by machine learning image information related to at least one of the user's facial expression and motion;
Determines joint coordinates according to securing connectivity for each joint candidate, and determines the motion information according to a change in the determined joint coordinates, and uses the determined motion information and predetermined motion information for each of the plurality of exercise devices. Counting the number of exercises by machine learning or determining a posture mismatch; and
When it is determined as the posture mismatch, generating coaching information related to posture correction
How a fitness server works.

Images