KR20220056967A - A climbing support system and method that guides the climbing path and evaluates climbing results based on ai - Google Patents

Abstract

The present invention relates to a climbing support method to enable a climber to efficiently perform climbing. According to the present invention, the climbing support method performed by a server comprises the following steps: recognizing hold information, which is information about positions and sizes of a plurality of holds included in a rock wall image received from an image sensor, by using a learned first machine learning module; receiving problem information about at least one climbing route designed on the basis of the hold information from a user; transmitting the received problem information to a projection apparatus; and evaluating a climbing image received from the image sensor by using a learned second machine learning module.

Description

A CLIMBING SUPPORT SYSTEM AND METHOD THAT GUIDES THE CLIMBING PATH AND EVALUATES CLIMBING RESULTS BASED ON AI

The present invention relates to a climbing support system and method for guiding a climbing route and evaluating a climbing result based on AI. Specifically, it uses a pre-learned machine learning model to guide a climber a climbing route and evaluate the climber's climbing result It relates to a climbing support system and method.

Sports climbing is a sport where you climb an artificial rock wall installed indoors by imitation of a natural rock wall. Sports climbing uses all the muscles of the body and consumes a lot of calories, and you can feel a sense of fun and achievement in the process of completing the climbing route (hereinafter, "problem") designed by the questioner.

The number of sports climbing centers in Korea is gradually increasing, and accordingly, the number of club members who enjoy sports climbing as a hobby is also gradually increasing.

An artificial rock wall is formed by combining various types of holds to an artificial wall that mimics a natural rock wall, and the question consists of a plurality of holds designated by the examiner. Climbers (hereinafter, "climbers") perform climbing by stepping on the hold designated by the examiner with their feet and holding them with their hands.

In general, climbers want to try different types of problems.

Accordingly, in order to meet the requirements of climbers in a limited space, the sports climbing center forms a large number of problems in one artificial rock wall, allowing climbers to climb.

In general, in order to distinguish a large number of problems formed in one artificial rock wall, holds constituting the same problem are formed in the same color, or band tapes of the same shape/color are attached to the holes constituting the same problem.

However, since a large number of problems are formed in one artificial rock wall, there is a problem that the climber has difficulty in finding the position of the next hold in the process of performing the problem.

Prior art document (Patent Publication No. 10-1586374) discloses a climbing system that informs a climber a climbing route through a projector. Specifically, the climbing system discloses a climbing system that informs a climbing route using a predetermined arrangement of a specific artificial rock wall or manually inputted coordinates.

However, in most sports climbing centers, the hold of the artificial rock wall is irregularly configured, and the position of the hold is changed periodically for the enjoyment of climbers in most cases. Accordingly, there may be a problem in that the human cost and time required to form a route for guidance are excessively increased.

Republic of Korea Patent Publication No. 10-1586374 (registered on November 21, 2019)

The present invention for solving the above problems is a climbing support system and method capable of recognizing hold information about the size and position of the hold of the artificial rock wall in the state in which the hold is already installed using a pre-learned machine learning model Its purpose is to provide

In addition, an object of the present invention is to provide a climbing support system and method capable of projecting an image to a hold that forms a climbing path based on the recognized hold information.

Another object of the present invention is to provide a climbing support system and method capable of recommending a climbing route suitable for a climber based on user information of the climber.

Another object of the present invention is to provide a climbing support system and method in which a climber can directly create and play a problem based on the recognized hold information.

In addition, an object of the present invention is to provide a climbing support system and method capable of evaluating a climber's climbing using a pre-trained machine learning model.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

Climbing support method according to an embodiment of the present invention for solving the above-mentioned problems, as a climbing support method performed by a server, the server, using the first machine learning module previously learned, received from the image sensor recognizing hold information, which is information about the positions and sizes of a plurality of holds included in the rock image; receiving, by the server, problem information about at least one climbing path previously designed based on the hold information from the user; transmitting, by the server, the received problem information to a projection device; and evaluating, by the server, the climbing image received from the image sensor using the second machine learning module previously learned.

The recognizing the hold information may include: receiving the rock wall image from the image sensor; recognizing hold information regarding the position and size of the hold included in the rock wall image by using the first machine learning module; and storing the hold information by matching with center information about the climbing center where the rock image was taken.

In addition, the center information includes at least one of the location of the climbing center where the rock wall image was captured and the location of the artificial rock wall where the rock wall image was captured.

In addition, the first machine learning module includes a hold recognition model learned through a plurality of hold recognition training data, and the hold recognition training data is a rock image with a rectangular box label corresponding to the size of the hold.

In addition, the step of receiving the problem information may include: transmitting the problem information to the user; and receiving selection problem information from the user, wherein the selection problem information is information about a climbing path selected by the user from among the at least one climbing path included in the problem information.

In addition, the problem information includes recommended problem information, and transmitting the problem information may include: receiving user information including at least one of height, weight, gender, proficiency, age, and wingspan from the user; and transmitting the recommendation problem information to the user based on the user information.

Also, in the transmitting of the recommended problem information, the similarity is determined by matching user information of another user who has previously climbed with the user information of the user, and the similarity is preset among the at least one climbing path. It is to transmit the climbing route performed by other users greater than or equal to the value as the recommended problem information.

The receiving of the problem information may include: transmitting the hold information to the user; receiving, from the user, custom problem information about the climbing path created by the user based on the hold information; and storing the custom problem information by matching with center information about the climbing center where the rock image was taken.

In addition, the climbing path created by the user is formed by the user selecting at least one hold constituting the climbing path from among the plurality of holds based on the position of the hold included in the hold information.

In addition, evaluating the climbing image may include: receiving the climbing image from the image sensor; identifying a skeleton vector in the received climbing image; performing a climbing evaluation based on the skeleton vector and the problem information using the second machine learning module; and matching and storing the result of the climbing evaluation with the user information and the problem information climbed by the user, wherein the user information is at least one of the user's height, weight, gender, proficiency, age, and wingspan. includes

In addition, the second machine learning module includes a motion evaluation model learned through a plurality of motion evaluation training data, the motion evaluation training data, the position of the hold included in the problem information; In the correct action, the distance between the skeleton vectors of both hands and both feet and the hold included in the question information; and information on the positions of the remaining skeleton vectors except for both hands and feet.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium for recording a computer program for executing the method may be further provided.

According to the present invention as described above, the climber can easily recognize the hold position through the image displayed on the hold constituting the problem. Therefore, the climber can focus on the problem without worrying about checking the hold position in the process of solving the problem. As a result, the probability of solving more problems using limited stamina is increased, and the climber can climb more efficiently.

In addition, in order to produce the image displayed on the hold, information on the position and size of the artificial rock wall is essential. Since the artificial rock wall is recognized based on machine learning, the artificial rock wall can be held only by taking a picture of the artificial rock wall. location and size information of can be obtained. As a result, it can be applied to an artificial rock wall already installed without any additional cost, and information on the position and size of the hold can be easily obtained even when the position of the hold is periodically rearranged.

In addition, the climber can directly create a problem based on the size and position information of the hold, and the created problem can be shared with other climbers. In this way, the climber can easily find the problem he wants to climb and perform the climb.

In addition, the degree of similarity with other climbers may be determined based on the climber's physical information, and a problem may be recommended based on the degree of similarity. Through this, the climber can perform climbing by being provided with a problem suitable for his/her physical ability.

In addition, a climbing image of a climber may be evaluated by the learned machine learning model. Since the evaluation is performed by the trained machine learning model, the climber's climbing can be evaluated by the image without the need to install a separate additional hardware device.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 and 2 are conceptual views showing the operating state of the climbing support method according to an embodiment of the present invention.
3 is a flowchart illustrating a specific process of a climbing support method according to an embodiment of the present invention.
4 is a flowchart illustrating a specific process of step S10 of FIG. 1 .
5 is a flowchart illustrating a detailed process of an embodiment of step S20 of FIG. 1 .
6 is a flowchart illustrating a detailed process of step S211 of FIG. 4 .
7 is a flowchart illustrating a detailed process of another embodiment of step S20 of FIG. 1 .
8 is a flowchart illustrating a specific process of step S40 of FIG. 1 .
9 is a block diagram showing a specific configuration of a climbing support system according to an embodiment of the present invention.
10 is a flowchart illustrating a specific learning process of the first machine learning module according to the present invention.
11 is a conceptual diagram illustrating an operation state of step S72 of FIG. 10 .
12 is a flowchart illustrating a specific learning process of the second machine learning module according to the present invention.
13 is a conceptual diagram illustrating an image of climbing a correct answer in which a skeleton vector is identified.
14 to 16 are conceptual views illustrating an operation state of step S83 of FIG. 12 .

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Before the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that, since the description of the term is for the purpose of helping the understanding of the present specification, it is not used in the meaning of limiting the technical idea of the present invention unless explicitly described as limiting the present invention.

The term “image” used below means that one image data or a plurality of different image data is continuously reproduced at a preset time interval.

The term "user" as used below means a person who performs climbing using the climbing support system according to the invention or a person who uses the climbing support system according to the invention to set up a climbing route.

The term “hold” used below refers to a structure that is coupled to an artificial rock wall, and is formed in various shapes so that a user who performs climbing can hold it by hand or be bright.

The term "problem" used below means a climbing path formed by a plurality of preset holds.

The term "user terminal" used below, as an electronic device capable of communicating with the server 10 , displays information received from the server 10 to the user, and the user requests information from the server 10 . and means a device capable of providing a user interface capable of transmitting a user's request to the server 10 . For example, the user terminal may be composed of a smart phone, a tablet PC, a notebook computer, and the like.

Hereinafter, with reference to FIGS. 1 and 2, a climbing support method according to an embodiment of the present invention will be described.

1. Description of the climbing support method according to an embodiment of the present invention

The climbing support method according to this embodiment is a method performed by the server 10 , and the image sensor 20 , the user terminal 30 and the projection device 40 communicatively connected to the server 10 . carried out by action. A specific configuration of the server 10 will be described in detail later with reference to FIG. 9 .

Climbing support method according to this embodiment includes the following process.

First, the image sensor 20 detects a rock image obtained by photographing an artificial rock wall, and transmits the detected rock image to the server 10, which is included in the rock image using a pre-learned machine learning model. The positions and sizes of the held holds are identified (S10).

In an embodiment, a rectangular box corresponding to the size of the hold is formed by the pre-trained machine learning model, and the size and position of the hold may be identified by the formed rectangular box.

Since the position of the hold can be automatically identified by the pre-learned machine learning model, the user can identify the information on the size and position of the hold included in the image of the rock just by photographing the artificial rock using the image sensor 20 . can do.

In the case of rearranging the hold of the artificial rock wall, there was a cumbersome need to input information on the size and position of the hold one by one in the prior art. However, according to the present invention, the size and position of the hold can be easily adjusted by shooting using the image sensor 20 can be identified.

In addition, the server 10 transmits, to the user terminal 30 , information on the size and position of the identified hold and problem information on the previously designed climbing path based on the size and position of the identified hold, and the user terminal 30 ) to receive the selected problem information or custom problem information produced by the user terminal 30 (S20).

The user terminal 30 may receive from the server 10 problem information about the previously designed and stored climbing path.

The user terminal 30 provides a user with a user interface (UI) through which the user can visually recognize and select problem information.

In one embodiment, the problem included in the problem information may be classified into beginner, intermediate, and advanced according to the difficulty level, and may be stored to match the name of the manufacturer and the name of the climbing center. Therefore, the user can check the problems that can be selected through the user terminal 30 by dividing by difficulty, and can check the production name and the climbing center name for each problem.

In addition, the user terminal 30 provides the user with a user interface (UI) through which the user can directly create a problem.

In an embodiment, the user terminal 30 may objectify the rock wall image including the hold information and display it to the user. In the objectified information, the sizes and positions of a plurality of holds are indicated. A user can form a custom problem by selecting a desired hold from a plurality of holds indicated in the objectified information.

In addition, in an embodiment not shown, the user may distinguish and designate a start hold corresponding to a start point, an end hold corresponding to an end point, and an intermediate hold forming a path between the start hold and the end hold.

In addition, in an embodiment not shown, the user can distinguish and designate a foot hold that must be touched using only the foot, a hand hold that must be touched using only the hand, and a general hold that can be touched using both the hand and the foot.

When the custom problem information created by the user or the problem information selected by the user is received, the server 10 transmits the received problem information to the projection device 40 (S30).

The projection device 40 projects an image corresponding to the received problem information on the artificial rock wall.

In one embodiment, the image corresponding to the received problem information is to display a hold display in a preset form on each hold constituting the climbing path included in the received problem information, and to display a path connecting the hold marks. can The hold mark may be in the form of a circle, and the path mark may be in the form of a straight line.

In an embodiment not shown, the hold display may be in the form of a hand or a foot.

The user can easily recognize the hold position of the problem through the image displayed on the projection device 40 to understand.

Therefore, the user can focus on the problem without worrying about confirming the hold position in the process of solving the problem.

As a result, the probability of solving more problems using limited physical strength is increased, and the user can climb more efficiently.

In addition, when the image is projected by the projection device 40, the image sensor 20 detects a climbing image for the process of the user solving a problem, the server 10 receives the detected climbing image, the received climbing The image is evaluated using a pre-trained machine learning model (S40).

In an embodiment, the pre-trained machine learning model may determine whether the user's skeleton vector matches the position of the hold in question using a pre-learned algorithm.

In an embodiment, the pre-trained machine learning model may divide and evaluate the user's climbing result for each start hold, middle hold, and end hold.

Referring to Figure 3, the process of the climbing support method described with reference to Figures 1 and 2 is shown.

(1) Description of the step (S10) of recognizing hold information, which is information about the positions and sizes of a plurality of holds included in the rock image received from the image sensor, by using the pre-learned first machine learning module

First, before generating a problem for a specific artificial rock wall, information about the location and size of holds installed on a specific artificial rock wall is recognized.

Referring to FIG. 4 , a detailed process of this step is shown.

First, the server 10 receives the rock wall image from the image sensor 20 (S11).

When the image sensor 20 detects the rock wall image, the rock wall image is transmitted from the image sensor 20 to the server 10 , and the server 10 receives the rock wall image from the image sensor 20 .

In an embodiment, the rock wall image is preferably taken at an angle at which all holds installed on the artificial rock wall can be identified. The rock wall image may be captured at an angle looking at the center of the artificial rock wall.

In addition, the server 10 recognizes the size and position of the hold included in the rock wall image by using the pre-learned first machine learning module (S12).

In an embodiment, the pre-trained first machine learning module includes a hold recognition model learned through a plurality of hold recognition training data, and the hold recognition training data is a rock image with a rectangular box label corresponding to the size of the hold. can In this regard, it will be described in detail later with reference to FIG. 11 .

When the position and size of the hold included in the received image information is recognized, the server 10 matches the hold-recognized rock wall image with the information of the climbing center and stores it (S13).

In one embodiment, the information of the climbing center includes the location of the climbing center where the rock image was taken and the location of the artificial rock wall where the rock image was taken.

Referring back to Figure 2, climbing support method according to the present invention includes the step (S20) of receiving the problem information from the user terminal (30).

(2) Description of the step (S20) of the server 10 receiving problem information about at least one climbing path designed based on the hold information from the user terminal 30

Referring to FIG. 5 , step S20 according to an embodiment includes a step of transmitting pre-designed problem information to the user terminal 30 ( S211 ) and a step of receiving selection problem information from the user terminal 30 ( S212 ). do.

The previously designed problem information includes at least one climbing path designed by other existing users using hold information of a specific artificial rock wall.

In one embodiment, the previously designed problem information is stored to match the name of the designer, the name of the climbing center and the location of the artificial rock wall. Accordingly, when the user specifies the name of the climbing center and the location of the artificial rock wall through the UI of the user terminal 30, the user may select at least one pre-designed climbing route for the specific artificial rock wall. In addition, the user may search for the designer's name and select at least one previously designed climbing route.

Referring to FIG. 6 , a detailed process of step S211 is illustrated.

The server 10 receives user information including at least one of the user's height, weight, gender, proficiency, age, and wingspan (length between fingertips when both arms are spread out) from the user terminal 30 . Receive (S2111).

In addition, the server 10 transmits the recommended problem information to the user terminal 30 based on the received user information (S2112).

For climbing, a preferred climbing route or a climbing route suitable for the user may vary depending on the user's height, weight, gender, skill level, age, and wingspan. For example, one climbing path might feel easier and more enjoyable for users with tall wingspans, while another climbing path might feel easier and more fun for users with shorter wingspans. there is.

In consideration of this, the present invention can provide recommended problem information about a climbing route that is suitable for a user and is expected to be more preferred by the user.

In an embodiment, the similarity may be determined by matching user information of a user who is currently climbing with user information of another user who has already climbed, and a user whose similarity is greater than or equal to a preset value may be selected. The similarity may be determined based on whether the gender matches, whether the skill level matches, and whether the differences in height, weight, age, and wingspan are within a preset error range.

The server 10 may transmit, to the user terminal 30 , information on the recommended problem regarding the climbing path that the user selected based on the similarity climbed.

In an embodiment, the server 10 may list the climbing paths climbed by the user selected based on the similarity in the order of the evaluation result score in the order of the highest score, and transmit it to the user terminal 30 . Regarding the evaluation result score, it will be described in detail later.

When the user selects a climbing path based on the problem information provided by the server 10 , the server 10 receives selection problem information about the climbing path selected by the user from the user terminal 30 ( S212 ).

In addition, referring to FIG. 7 , the step S20 according to another embodiment includes the step of transmitting hold information to the user terminal 30 ( S221 ), and receiving the custom problem information created by the user based on the hold information ( S222 ). ) and storing the custom problem information by matching with the center information about the climbing center (S223).

First, the server 10 transmits the hold information on the position and size of the hold included in the artificial rock wall to the user terminal 30 (S221).

In one embodiment, when the user selects a specific climbing center and artificial rock wall through the UI of the user terminal 30 and selects a question paper, the server 10 objectifies the hold information of the selected artificial rock wall to the user terminal 30 ) is sent to

In the objectified information, the sizes and positions of a plurality of holds are indicated. A user can form a custom problem by selecting a desired hold from a plurality of holds indicated in the objectified information.

In addition, in an embodiment not shown, the user may distinguish and designate a start hold corresponding to a start point, an end hold corresponding to an end point, and an intermediate hold forming a path between the start hold and the end hold.

In addition, in an embodiment not shown, the user can distinguish and designate a foot hold that must be touched using only the foot, a hand hold that must be touched using only the hand, and a general hold that can be touched using both the hand and the foot.

When the production of the climbing route is completed, the server 10 receives custom problem information about the climbing route created by the user's selection from the user terminal 30 (S222).

When the custom problem information is received, the server 10 stores the custom problem information by matching with the center information about the climbing center (S223).

(3) Description of the step (S30) of the server 10 transmitting the received problem information to the projection device 40

The projection device 40 projects an image corresponding to the received problem information on the artificial rock wall.

In one embodiment, the image corresponding to the received problem information is to display a hold display in a preset form on each hold constituting the climbing path included in the received problem information, and to display a path connecting the hold marks. can The hold mark may be in the form of a circle, and the path mark may be in the form of a straight line.

In an embodiment not shown, the hold display may be in the form of a hand or a foot.

(4) Description of the step (S40) of the server 10 evaluating the climbing image received from the image sensor 30 using the second machine learning module previously learned

Referring to FIG. 8 , a detailed process of step S40 is illustrated.

First, when an image is projected by the projection device 40 , the image sensor 30 starts photographing a user who climbs on the artificial rock wall. The server 10 receives the climbing image detected by the image sensor from the image sensor 30 (S41).

In addition, the server 10 identifies a skeleton vector (Skeleton Vector) by using the received climbing image.

In an embodiment, the skeleton vector includes coordinates of the user's hand, elbow, shoulder, knee, and foot.

In addition, the server 10 uses the pre-learned second machine learning module to perform climbing evaluation based on the skeleton vector and problem information ( S43 ).

In an embodiment, the second machine learning module includes a motion evaluation model learned through a plurality of motion evaluation training data.

The motion evaluation learning data is data that labels the correct answer climbing image of a performer who has climbed the correct answer for a specific problem. Specifically, after the labeler divides and divides the correct climbing image by hold, it labels that it may correspond to the motion evaluation learning data.

The motion evaluation learning data includes information on the position of the hold in the correct motion for each hold and the distance between the skeleton vectors of both hands and both feet, and information on the positions of the skeleton vectors excluding both hands and feet.

Accordingly, whether the operation is properly performed may be evaluated for each hold constituting the problem. Accordingly, the evaluation can be performed more specifically and accurately.

The correct action may be composed of a plurality of different actions, and the second machine learning module may be classified and learned according to user information.

For example, after generating motion evaluation learning data by processing only the correct motion of a woman whose height is 165 to 170 cm, the second machine learning module may be trained using the motion evaluation learning data. Through this, the accuracy of evaluation may be improved.

In the second machine learning module, the distance between the vectors of both hands and feet among the skeleton vectors of the climbing image and the hold included in the problem information, and the positions of the skeleton vectors in other parts except for both hands and feet, are the correct answers learned through the motion evaluation learning data. It is determined whether it can be judged as an action.

When the evaluation result is derived, the server 10 matches and stores the evaluation result with the user information and the problem information climbed by the user (S44).

Also, the evaluation result may be transmitted to the user terminal 30 and provided to the user.

In an embodiment, the evaluation result may include information on whether the problem succeeds, the number of the last hold performed in case of failure, and the like.

2. Description of the climbing support system according to an embodiment of the present invention

Referring to Figure 9, the configuration of the climbing support system for performing the above-described climbing support method is shown.

The climbing support system according to this embodiment includes a server 10 , an image sensor 20 , a user terminal 30 and a projection device 40 .

The image sensor 20 , the user terminal 30 , and the projection device 40 are configured to communicate with the server 10 .

In an embodiment, the server 10 is configured to exchange information with communication-connected devices, calculate other information using the received information, and store the received or calculated information. For example, the server 10 may be configured using Amazon Web Service (AWS). However, the present invention is not limited thereto.

The server 10 includes a hold recognition unit 11 , a UI providing unit 12 , an image providing unit 13 , an evaluation unit 14 , and a database unit 15 .

The hold recognition unit 11 and the evaluation unit 14 are configured to identify a hold using a pre-trained machine learning model and evaluate the user's climbing image.

In an embodiment, the hold recognition unit 11 and the evaluation unit 14 may be configured as edge devices located outside the server 10 . For example, the hold recognition unit 11 and the evaluation unit 14 may be configured with a Google Coral Dev Board that performs learning by TensorFlow and has Tensor Processing Units (TPUs) embedded therein. However, the present invention is not limited thereto.

The UI providing unit 12 is configured to provide information stored in the database unit 15 to the user terminal 30 , and to store information received from the user terminal 30 in the database unit 15 . In an embodiment, the UI providing unit 12 may be configured with Amazon EC2. However, the present invention is not limited thereto.

The database unit 15 is configured to store information calculated by the server 10 and various types of information received by the server 10 . In an embodiment, the database unit 15 may be configured with Amazon S3 and Amazon RDS. However, the present invention is not limited thereto.

(1) Description of the hold recognition unit 11

The hold recognition unit 11 includes a rock wall image receiving module 111 and a first machine learning module 112 .

The rock wall image receiving module 111 receives the rock wall image from the image sensor 20 .

The first machine learning module 112 uses the received rock wall image to identify the position and size of the hold included in the rock wall image. The first machine learning module 111 has been described above with the method of climbing support of the present invention, it is replaced.

The hold recognition unit 11 may transmit the recognized information received from each module to the database unit 15 .

(2) Description of the UI providing unit 12

The UI providing unit 12 includes a user information management module 121 , a problem providing module 122 , a custom problem providing module 123 , and an evaluation providing module 124 .

The user information management module 121 is configured to receive user information from the user terminal 30 . User information has been described above with the method of climbing support of the present invention, instead of this.

The problem providing module 122 is configured to transmit the problem information stored in the database unit 15 to the user terminal 30 .

In one embodiment, when a request to provide problem information for a specific artificial rock wall of a specific climbing center is received from the user terminal 30, the problem providing module 122 provides problem information corresponding to the request to the database unit 15 is transmitted to the user terminal 30 . The problem information may be provided after being classified by difficulty level.

In addition, in an embodiment, when a request to provide problem information for a specific designer is received from the user terminal 30 , the problem providing module 122 provides problem information corresponding to the request from the database unit 15 to the user terminal (30).

In addition, in an embodiment, the problem providing module 122 transmits, to the user terminal 30 , recommended problem information regarding a climbing route performed by another user whose similarity with the user information of the user who requested the problem provision is equal to or greater than a preset value. can be transmitted

In addition, according to an embodiment, the problem providing module 122 may arrange the climbing paths included in the recommended problem information in the order of the evaluation result, and provide it to the user terminal 30 .

In an embodiment, the value of the evaluation result may be measured to be higher as the time required for climbing is shorter, and the number of holds that have succeeded in climbing among all holds of the climbing path is greater.

In addition, the problem providing module 122 receives the problem information about the selected climbing path from the user terminal (30).

When a question request for a specific artificial rock wall of a specific climbing center is received from the user terminal 30, the problem custom providing module 123 objects information about the positions and sizes of all holds installed on the artificial rock wall to the user terminal ( 30) is sent.

In addition, when information on the hold selection is received from the user terminal 30 , the problem custom providing module 123 transmits information corresponding to the information on the hold selection to the user terminal 30 . Accordingly, the hold selected by the user may be displayed on the display of the user terminal 30 .

In an embodiment, the information on the hold selection may include information on designation of a start hold, an intermediate hold, and an end hold, and information on the designation of a hand hold, a foot hold, and a normal hold.

In addition, the problem custom providing module 123 receives problem information about the climbing path created by the user from the user terminal 30 .

The evaluation providing module 124 receives information about the evaluation request from the user terminal 30, transmits the evaluation request to the evaluation unit 14 in response thereto, and returns the evaluation result evaluated by the evaluation unit 14 to the user terminal ( 30) is configured to transmit.

Various information received and calculated by each module of the UI providing unit 12 may be transmitted to and stored in the database unit 15 .

(3) Description of the image providing unit 13

The image providing unit 13 is configured to transmit the problem information received by the problem providing module 122 and the problem custom providing module 123 to the projection device 40 .

(4) Description of the evaluation unit 14

The evaluation unit 14 includes a climbing image receiving module 141 , a problem receiving module 142 , a motion detection module 143 , and a second machine learning module 144 .

The climbing image receiving module 141 is configured to receive a climbing image from the image sensor 20 , and the problem receiving module 142 is configured to receive problem information about the climbing path received from the user terminal 30 .

In addition, the motion detection module 143 is configured to identify the user's skeleton vector included in the received climbing image.

In addition, the second machine learning module 144 is configured to evaluate the user's climbing based on the image and the problem information identified by the skeleton vector in the motion detection module 143 . The second machine learning module 144 has been described above along with the method of climbing support of the present invention, which is in lieu of.

The evaluation unit 14 may transmit information received from each module, sensed information, and evaluated information to the database unit 15 .

In some embodiments, the server 10 may include fewer or more components than the components shown in FIG. 9 .

The climbing support system according to the embodiment of the present invention described above is different from the climbing support method and the category of the invention described with reference to Figs.

3. Description of the step (S70) in which the first machine learning module 112 learns

Referring to FIG. 10 , a detailed process of step S70 is illustrated.

First, a plurality of different artificial rock data is transmitted from the database unit 15 to the first labeling device 51 . The artificial rock data may be transmitted from a storage device other than the database unit 15 .

In one embodiment, the first labeling device 51 may be a device provided with means for the first labeling operator to perform a labeling operation. For example, the first labeling device 51 may be an electronic device in which a labeling program is installed.

In an embodiment, the artificial rock wall data includes image information of the artificial rock wall in a state in which the hold is installed.

When the artificial rock data is received by the first labeling device 51, the first labeling worker labels the hold included in the artificial rock data using the first labeling device 51 (S72).

Referring to FIG. 11 , the labeling operation performed by the first labeling operator is shown.

The first labeling operator may apply a square box label to the hold included in the artificial rock wall image by using the first labeling program.

Each rectangular box label can be set to a minimum size that can contain a corresponding hold.

Referring back to FIG. 10 , the labeled artificial rock data is transmitted from the first labeling device 51 to the first machine learning module 112 ( S73 ).

The first machine learning module 112 performs learning using the labeled artificial rock data (S74).

In an embodiment, transfer learning may be used to learn artificial rock data, and an SSD ResNet50 v1 model may be used. However, the present invention is not limited thereto.

4. Description of the second machine learning module 144 learning step (S80)

Referring to FIG. 12 , a detailed process of step S80 is illustrated.

First, a plurality of different correct climbing images for which the skeleton vector is identified in the database unit 15 are transmitted to the second labeling device 52 . The artificial rock data may be transmitted from a storage device other than the database unit 15 .

In one embodiment, the second labeling device 52 may be a device provided with means for the second labeling operator to perform a labeling operation. For example, the second labeling device 52 may be an electronic device in which a labeling program is installed.

Referring to FIG. 13 , an example of a correct answer climbing image in which a skeleton vector is identified is shown.

In the correct climbing image, the skeleton vector (1a) corresponding to the hands and feet of the performer performing the climbing, the skeleton vector (1b) corresponding to parts other than both hands and feet, and a rectangular box indicating the position of the hold corresponding to the climbing path A label (2) is included.

In one embodiment, the second labeling device 52 may receive the user information of the correct climbing performer (S82).

When the artificial rock data is received by the second labeling device 52, the second labeling worker uses the second labeling device 52 to label data that can be used as motion evaluation learning data in the correct answer climbing image (S83).

That is, the second labeling operator assigns a label to the image in which the correct answer operation is performed for each hold among the images included in the correct answer climbing image.

14 to 16, the process of the labeling operation of the second labeling operator is shown.

Referring to FIG. 14 , the second labeling operator confirms whether at least one of the skeleton vectors corresponding to both hands and feet of the performer is located in every start hold. If at least one of the skeleton vectors corresponding to both hands and feet of the performer is located in every start hold, the second labeling operator labels the corresponding image as motion evaluation learning data.

Referring to FIG. 15 , the second labeling operator confirms whether at least one of the skeleton vectors corresponding to both hands and feet of the performer is located in the middle hold of the designated number.

In addition, the second labeling operator confirms whether at least one of the skeleton vectors corresponding to both hands and feet of the performer is located in the previous hold of the intermediate hold of the designated number.

When the above conditions are satisfied, the second labeling operator labels the image as motion evaluation learning data.

Referring to FIG. 16 , the second labeling operator checks whether at least two of the skeleton vectors corresponding to both hands and feet of the performer are located in the end hold. If this is satisfied, the second labeler labels the image as motion evaluation learning data.

Referring back to FIG. 12 , the second labeling device 52 transmits the labeled data to the second machine learning module 144 ( S84 ), and the second machine learning module 144 uses the received data to learn motion evaluation. Learning is performed using data (S85).

The method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The above-described program is C, C++, JAVA, machine language, etc. that a processor (CPU) of the computer can read through a device interface of the computer in order for the computer to read the program and execute the methods implemented as a program It may include code (Code) coded in the computer language of Such code may include functional code related to a function defining functions necessary for executing the methods, etc., and includes an execution procedure related control code necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. there is. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and an optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected to a network, and a computer-readable code may be stored in a distributed manner.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains know that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: Server
11: Hold recognition unit
111: rock wall image receiving module
112: first machine learning module
12: UI provider
121: user information management module
122: problem providing module
123: Problem custom provided module
124: evaluation providing module
13: video providing unit
14: Evaluation Department
141: climbing video receiving module
142: problem receiving module
143: motion detection module
144: second machine learning module
15: database unit
20: image sensor
30: user terminal
40: projection device
51: first labeling device
52: second labeling device

Claims (12)

As a climbing support method performed by a server,
recognizing, by the server, hold information, which is information about positions and sizes of a plurality of holds included in a rock image received from an image sensor, using a pre-learned first machine learning module;
receiving, by the server, problem information about at least one climbing path previously designed based on the hold information from the user;
transmitting, by the server, the received problem information to a projection device; and
Comprising the step of evaluating, by the server, the climbing image received from the image sensor using a second machine learning module previously learned,
How to support climbing. According to claim 1,
Recognizing the hold information comprises:
receiving the rock wall image from the image sensor;
recognizing hold information regarding the position and size of the hold included in the rock wall image by using the first machine learning module; and
Comprising the step of matching and storing the hold information with center information about the climbing center where the rock image was taken,
How to support climbing. 3. The method of claim 2,
The center information includes at least one of the location of the climbing center where the rock wall image was taken and the location of the artificial rock wall where the rock wall image was taken,
How to support climbing. 3. The method of claim 2,
The first machine learning module includes a hold recognition model learned through a plurality of hold recognition training data,
The hold recognition learning data is a rock wall image with a rectangular box label corresponding to the size of the hold,
How to support climbing. According to claim 1,
Receiving the problem information includes:
transmitting the problem information to the user; and
receiving selection problem information from the user;
The selection problem information is information about a climbing path selected by the user from among the at least one climbing path included in the problem information,
How to support climbing. 6. The method of claim 5,
The problem information includes recommended problem information,
Transmitting the problem information includes:
receiving user information including at least one of height, weight, gender, skill level, age, and wingspan from the user; and
Comprising the step of transmitting the recommended problem information to the user based on the user information,
How to support climbing. 7. The method of claim 6,
The step of transmitting the recommended problem information is,
Matching the user information of the user with the user information of another user who has previously performed climbing determines the degree of similarity,
Of the at least one climbing route, a climbing route performed by another user having the similarity greater than or equal to a preset value is transmitted as the recommended problem information,
How to support climbing. According to claim 1,
Receiving the problem information includes:
transmitting the hold information to the user;
receiving, from the user, custom problem information about the climbing path created by the user based on the hold information; and
Comprising the step of matching and storing the custom problem information with center information about the climbing center where the rock image was taken,
How to support climbing. 9. The method of claim 8,
The climbing route created by the user is,
Based on the position of the hold included in the hold information, the user selects at least one hold constituting a climbing path among the plurality of holds and is formed,
How to support climbing. According to claim 1,
The step of evaluating the climbing image,
receiving the climbing image from the image sensor;
identifying a skeleton vector in the received climbing image;
performing a climbing evaluation based on the skeleton vector and the problem information using the second machine learning module; and
Comprising the step of matching and storing the result of the climbing evaluation with the problem information and user information climbed by the user,
The user information includes at least one of the user's height, weight, gender, skill level, age, and wingspan,
How to support climbing. 11. The method of claim 10,
The second machine learning module includes a motion evaluation model learned through a plurality of motion evaluation learning data,
The motion evaluation learning data is,
the position of the hold contained in the problem information;
In the correct action, the distance between the skeleton vectors of both hands and both feet and the hold included in the question information; and
Including information about the positions of the skeleton vectors other than both hands and feet,
How to support climbing. In combination with a computer, which is hardware, stored in a medium for executing the method of any one of claims 1 to 10,
program.

Images