KR20230124796A - System for providing identification code based climbing solution video streaming service - Google Patents

Abstract

A system for providing an identification code-based climbing solution video service is provided, and after scanning an identification code attached to at least one starting point of climbing, a user terminal and at least one starting point outputting a solution image pre-mapped to the identification code and stored therein A database unit that maps and stores solution images guiding the coordinates, at least one identification code, and at least one attitude, action, and route to be taken at the start point, and when the user terminal scans the identification code, the user terminal identifies the identification code. It includes a video service providing server including a transmission unit that transmits a pre-mapped and stored solution video and a control unit that controls a user terminal to play the solution video.

Description

Identification code-based climbing solution video service providing system {SYSTEM FOR PROVIDING IDENTIFICATION CODE BASED CLIMBING SOLUTION VIDEO STREAMING SERVICE}

The present invention relates to an identification code-based climbing solution video service providing system, and provides a platform that provides a solution video guiding a climbing posture, motion, and route when an identification code attached to a climbing start point is scanned.

Sport climbing is a sport that has been modernized so that you can safely enjoy the experience of rock climbing on a natural rock wall anywhere. In sport climbing, artificial rock walls are used instead of natural rocks. An artificial rock wall refers to a rock wall configured to be held by hand or stepped on by placing and fixing plywood or plastic panels so as to be reminiscent of a natural rock wall and attaching an artificial hold. The user sees the slope of the artificial rock wall and the attached hold, etc., and climbs while finding a route that can be climbed efficiently. Sport climbing is divided into outdoor and indoor. Outdoor sports climbing is climbing by constructing an artificial rock wall on the outer wall of a building or an artificial structure in an outdoor space. It requires skill. On the other hand, indoor sports climbing consists of climbing an artificial rock wall in an indoor space. In indoor sports climbing, because the indoor space is limited, the artificial rock wall is composed of a large area rather than a high structure.

At this time, a method of guiding the optimal route by attaching a sensor to a hold in rock climbing or identifying a climber's movement path and then guiding the position, posture and motion to go to with a projector has been researched and developed. In this regard, Korea registration as a prior art In Patent No. 10-1985963 (published on Jun. 4, 2019) and Korean Patent Registration No. 10-1586374 (published on Jan. 19, 2016), contact of a climber with the hold is sensed, and when the sensing information is input It is transmitted to the server along with the identification code, and the information on the climbing route on the artificial rock wall is stored and guided by including the identification code, and the climbing route that connects the hold that the climber has to climb is displayed and provided. A configuration for providing an image on a screen to efficiently teach climbing by recognizing a location and comparing the recognized location with a climbing reference path is disclosed.

However, in the former case, each hold must have a contact sensor. Due to the nature of climbing, the hold is held and the load is loaded, so the contact sensor itself is often damaged due to a load being caught. It may vary depending on each situation or climber, and even if a video is provided, it is not easy to watch the screen while hanging from an artificial rock wall. Therefore, it is required to research and develop a platform that attaches an identification code indicating the starting point and provides a solution image that guides the posture, motion, and path for climbing at the starting point.

One embodiment of the present invention attaches an identification code indicating the starting point, provides a solution image that guides the attitude, motion and route for climbing at the starting point, displays the identification code and the location of each starting point, , By building a database to map and store solution images to identification codes, users are provided with solutions on what posture, motion, and route to go at each starting point. Since AR content is augmented and displayed, it is possible to provide an identification code-based climbing solution video service providing system that can intuitively grasp the route, posture, and motion. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention scans an identification code attached to at least one starting point of climbing, and then outputs a solution image previously mapped to the identification code and stored. A database unit that maps and stores the coordinates of the terminal and at least one starting point, at least one identification code, and a solution image guiding the posture, motion, and route to be taken at the at least one starting point, and the user terminal scans the identification code. In this case, a video service providing server including a transmission unit for transmitting a stored solution video mapped to an identification code to a user terminal and a control unit for controlling the user terminal to play the solution video.

According to any one of the above-described problem solving means of the present invention, an identification code indicating the start point is attached, a solution image for guiding the posture, motion, and path for climbing at the start point is provided, and the identification code and each start By building a database that displays the location of the point, maps the solution image to the identification code, and stores it, it provides the user with a solution for what posture, motion, and route to go at each starting point, and screens when using smart glasses. AR content is augmented and displayed on the glass itself without looking at it, so you can intuitively grasp the route, posture, and movement.

1 is a diagram for explaining an identification code-based climbing solution video service providing system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a video service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which an identification code-based climbing solution video service is implemented according to an embodiment of the present invention.
5 is an operational flowchart for explaining a method of providing an identification code-based climbing solution video service according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining an identification code-based climbing solution video service providing system according to an embodiment of the present invention. Referring to FIG. 1 , an identification code-based climbing solution video service providing system 1 may include at least one user terminal 100, a video service providing server 300, and at least one wearable device 400. . However, since the identification code-based climbing solution video service providing system 1 of FIG. 1 is only one embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to a video service providing server 300 through a network 200 . Also, the video service providing server 300 may be connected to at least one user terminal 100 and at least one wearable device 400 through the network 200 . Also, at least one wearable device 400 may be connected to the video service providing server 300 through the network 200 .

Here, the network refers to a connection structure capable of exchanging information between nodes such as a plurality of terminals and servers, and examples of such networks include a local area network (LAN) and a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communications networks, telephone networks, and wired and wireless television communications networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi , Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( A Near-Field Communication (Near-Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but not limited thereto.

In the following, the term at least one is defined as a term including singular and plural, and even if at least one term does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, the singular or plural number of each component may be changed according to embodiments.

At least one user terminal 100 may be a terminal that scans an identification code of a starting point using a web page, app page, program, or application related to an identification code-based climbing solution video service, and outputs a solution video. When the user terminal 100 is linked with the wearable device 400, after receiving data sensed from the wearable device 400, the user terminal 100 transmits the data to the video service providing server 300, and the received data from the video service providing server 300 It may be a terminal that transmits and outputs data to the wearable device 400 .

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that ensures portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

The video service providing server 300 may be a server that provides an identification code-based climbing solution video service web page, app page, program or application. And, the video service providing server 300 is a server that maps and stores the coordinates of at least one starting point, at least one identification code, and a solution image guiding the attitude, motion, and route to be taken at the at least one starting point. can In addition, the image service providing server 300 may be a server that transmits the stored solution image mapped to the identification code to the user terminal 100 when the identification code is scanned by the user terminal 100 . In addition, when data is input from the wearable device 400, the video service providing server 300 analyzes it and provides a solution video corresponding to the data to the user terminal 100 so that the wearable device 400 can output the video. It may be a server that allows

Here, the video service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser.

At least one wearable device 400 outputs data from the video service providing server 300 via the user terminal 100 using a web page, app page, program or application related to the climbing solution video service based on the identification code , It may be a device that transmits the detected data to the video service providing server 300 via the user terminal 100 again. The data at this time may be voice, sound, video, vibration, and the like, and the video may include VR, AR, XR, MR, and the like. In this case, at least one wearable device 400 may include smart glasses.

Here, at least one wearable device 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one wearable device 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one wearable device 400 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

2 is a block diagram illustrating a video service providing server included in the system of FIG. 1, and FIGS. 3 and 4 are an embodiment in which an identification code-based climbing solution video service is implemented according to an embodiment of the present invention. It is a drawing for explaining.

Referring to FIG. 2, the video service providing server 300 includes a database builder 310, a transmitter 320, a controller 330, an AR solution provider 340, a posture and motion correction unit 350, and a real-time generation It may include a unit 360, a solution providing unit 370 and a deep learning analysis unit 380.

An identification code-based climbing solution in which the video service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100 and at least one wearable device 400 When transmitting a video service application, program, app page, web page, etc., at least one user terminal 100 and at least one wearable device 400 are identified code-based climbing solution video service applications, programs, app pages, You can install or open web pages, etc. Also, the service program may be driven in at least one user terminal 100 and at least one wearable device 400 by using a script executed in a web browser. Here, the web browser is a program that allows users to use the web (WWW: World Wide Web) service, and means a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language). For example, Netscape , Explorer, Chrome, and the like. In addition, an application means an application on a terminal, and includes, for example, an app running on a mobile terminal (smart phone).

Before describing FIG. 2, the basic concept of screen climbing will be described. Things described below will not be redundantly described in FIG. 2 .

<Screen Climbing>

Screen climbing means that climbing is progressed by a computer system equipped with conditions to be able to interact with the user for indoor sports climbing. The artificial rock wall becomes a screen and is expressed to the user as a real natural rock wall, and can also be expressed in other virtual exotic spaces. The user climbs by viewing the route displayed on the screen and hold projected on the artificial rock wall. The indoor artificial rock closet is a suitable environment for convergence of ICT technology. The outdoor artificial rock closet is not easy to install computer equipment and is affected by the weather. The images on the screen do not appear clearly due to the curvature of the rock wall and natural light. On the other hand, the indoor artificial rock closet is easy to install equipment such as computers, beam projectors, and cameras, and is not affected by the weather. The curve of the artificial rock wall is also flat, so it is suitable for use as a screen.

Screen climbing creates a climbing experience with the effect of actual rock climbing, which enhances the user's experience unlike simulation of an artificial environment. Nintendo Wii, which is typically used in a simulation environment, plays with movements close to real life while taking advantage of the characteristics of a remote controller. In a rock climbing game on the Nintendo Wii, there is an act of climbing a wall, but no climbing experience. Compared to actual climbing, the user's immersion is not high. However, screen climbing maximizes the user's immersion because both the climbing action and the experience occur using an experience-based physical interface called a rock wall.

Climbing route setting is important in climbing learning. Climbing route settings can explain the characteristics or risk of the route, and induce learning from low difficulty to high level according to the grade. Therefore, climbing beginners can learn systematically by choosing the lowest possible climbing route and increasing the climbing level as they gradually become accustomed to the difficulty. Climbing routes can be expressed as grades, and the most representative expression method uses the Yosemite Decimal System (YDS). Indoor sport climbing uses a decimal grading system of five degrees. A grade of 5.0 to 5.7 is where most beginners start climbing, and more experienced climbers find it rather easy. Grades from 5.8 to 5.9 may be difficult for beginners, but they are a level that can be climbed once you get used to it. A grade of 5.10 is a level that can only be reached with regular climbing practice. From this level, the difficulty is further subdivided into a, b, c, and d. Grades 5.11 to 5.15 are advanced climbers' territory and require effort and talent.

Climbing routes are determined by the Route Setter. A route setter is a person who makes a path on an artificial rock wall for athletes or climbers to climb. The route setter sets up a random climbing route and climbs it himself to classify the route. Usually, in the case of low climbing difficulty, the proportion of jug holds that are easy to hold by hand is high, and in the case of high climbing difficulty, artificial holds that are difficult to support by hand such as slopers or crimp holds are used. In indoor sport climbing, the climbing route is marked with numbered stickers next to the holds. The user climbs by following the numbers on the stickers for systematic climbing learning. In the case of an artificial rock wall where the climbing route is not displayed, the trainer guides the user to climb by pointing the hold to the user using a pointer or a laser pointer.

It is difficult to change the route in real time in the method of attaching number stickers to inform climbing routes, and users may be confused if many stickers are used to inform various routes. To solve this phenomenon in the indoor climbing gym, the position of the hold is periodically relocated and the climbing route is re-established. However, climbing route resetting is a long cycle and requires resources of time and labor. Accordingly, the user has a problem in that it is difficult for the user to experience various routes by himself and the user climbs in a monotonous course until the route is reset. Also, if the hold is in a high position or blocked by another hold, there is a limit to the trainer's ability to direct the hold.

In screen climbing, a climbing route with an interactive artificial rock wall is mapped and displayed on the screen according to the user's request. The user and the trainer use an application on a tablet or mobile device, the user selects a route suitable for his climbing difficulty, and the trainer selects a hold to instruct. Depending on the selection, the climbing route and instruction hold appear on the screen. This method does not confuse the user because the climbing route display is presented in one consistent way, and the route changes in real time. Since the user experiences various routes through route selection by himself, the monotony felt in the existing sports climbing is resolved. In addition, even if the hold is at a high position or is blocked by another hold, the limit of indicating the route is overcome by projecting using a beam projector.

In Screen Climbing, not only climbing route mapping but also various contents are provided to users. By recording the user's climbing route selection information and recording the climbing process, it is provided to the user as learning content. Climbing route selection information is recorded in the application and provides statistical data on a daily basis. Based on this, the user selects a level of difficulty suitable for the current climbing and leads them to climb. In addition, a timer is set to compete for climbing records between users or when mapping a climbing route, and the route automatically disappears after a certain period of time, and the next route appears as game-like content.

Referring to FIG. 2 based on the above-described basic concept, the database integrator 310 guides the coordinates of at least one starting point, at least one identification code, and the attitude, motion, and route to be taken at the at least one starting point. The solution image can be mapped and saved. Here, the posture, motion, and route may be a solution video taken directly by a person, but after creating a virtual person like a simulation, taking a posture or performing an action and moving each route, the simulation is performed as a solution video can also be shown as That is, in order to create a motion and posture for climbing, a climbing posture and motion may be created using inverse kinematics in a virtual 3D space. Generating or specifying specific motions for the movement of virtual characters is collectively referred to as motion control technology. It can be divided into basic methods. Kinematics-based techniques are further divided into forward kinematics (FK)-based techniques and inverse kinematics-based techniques.

The motion capture technique is a technique used to create animations by attaching spatial position (XYZ) and posture sensors called trackers to the human body to digitally convert information on changes in human posture and motion. The motion capture technique has the advantage of being able to implement complex postures or fast motions as real-time animations as the values of each joint for generating motions are provided in real time from the sensor. difficult. In one embodiment of the present invention, posture and motion can be created using motion control based on inverse kinematics. The inverse kinematics-based technique is the opposite of forward kinematics, and is a technique that recalculates the state of a joint based on a target point. Since it is easy to recalculate joint values for postures and motions according to various constraint conditions, various modified motions are possible.

Accordingly, in one embodiment of the present invention, a sports climbing posture may be generated through a virtual person using inverse kinematics in a digital 3D space, and motion may be generated using a sports motion procedure. Expert advice and actual measurement data can be used for the distance of hands or feet that can be placed to resolve variables related to climbing ability, and can be limited to a range of maximum and minimum values.

<Create Posture>

As a prerequisite for creating actual postures and motions, variables such as high-level motions that are difficult for beginners or ordinary people to actually take are excluded, and differences in climbing capabilities that come from individual differences also use expert advice, actual survey data, and Statistics Korea data. Therefore, the deviation can be minimized by obtaining the minimum and maximum distances that the virtual character can climb.

<Create basic posture>

In climbing, the basic posture is a half-sitting posture, unlike the normal standing posture, and the center of gravity is also very different from the general posture. In one embodiment of the present invention, an inverse kinematics function (IK Function) based on inverse kinematics for multi-joint body or human body deformation may be used. Inverse kinematics is the opposite concept of forward kinematics and is a method of obtaining the position or motion of each joint from the position or motion of the end point of the joint body. After creating both hands and feet at designated positions, the basic posture is created by correcting the position value of the body calculated according to the position values of the hands and feet to the basic position value of the body using actual measurement data.

<Creating body position>

An important part in creating a posture is creating a body position. This is because it causes many problems such as not being able to hold an impossible posture or hold depending on the position of the body. Like a line connecting the position of one hand and the position of one foot on the diagonal, the body position is calculated through the intersection of the two lines connecting the position of the other hand and the diagonal foot, and the expert's advice and actual data are calculated by The body position may be created by setting the calculated body position value as an offset value.

<Posture Correction>

The biggest problem in posture generation using inverse kinematics is that each joint can have more than one value for the end point, so it can have a different value than the desired value. In other words, wrong motions such as twist are generated due to wrong values of the joints of the virtual person's hands or feet. In order to solve this problem, the center of gravity of the basic posture is obtained using the actual measurement data and the data of the National Statistical Office, and the joint values are limited to the maximum and minimum values. If each joint value exceeds the error range, it can be recalculated to create the correct posture. .

<create action>

Basically, there are various methods for generating motions of virtual characters. In an embodiment of the present invention, inverse kinematics-based interpolation among interpolation methods for deformation of a multi-joint body or human body may be used. In addition, since basic motion procedures are established for movement in climbing, it is possible to create motions similar to actual climbing by understanding the basic motion procedures and sequentially creating motions using virtual characters in 3D space.

<motion interpolation>

Motion generation uses a spherical linear interpolation method, one of the interpolation methods. Interpolation is the process of estimating from known values a value that lies between the values of known points. Interpolation between points in the form of a line is called linear interpolation, and interpolation along a circular arc is called spherical interpolation. In one embodiment of the present invention, the motion of the posture created in the above-described posture generation and the posture of the next position may be generated through spherical interpolation. When generating motion using commonly used linear interpolation, object interference between distances moving in 3D space, for example, a hand or foot gets caught in a large volume hold between holds or holds There are other holds in the hold, causing problems with the hand or foot passing through the hold. Along with spherical interpolation, the above problem can be solved by changing the rotation value of interpolation according to the specific situation.

<Setting the position of the body>

An important part in creating motion is also setting the body location. In the case of simply interpolating the position between the body postures calculated by the position of the hands and feet, the motion interpolated position of the hands and feet cannot be reflected in the resulting change in the position of the body, thus creating artificial motion due to the constantly interpolated body position regardless of this Therefore, there is a big difference from the actual human motion. According to the moving hands and feet, the intersection point of the two lines mentioned in body position creation is recalculated in real time and used to interpolate the body position. In addition, if the two lines do not intersect depending on the moving position of the hand or foot, the other line segment is projected onto the line connecting the moving hand and foot, recalculated, and reflected on the body position to create motion. .

<Body tilt setting>

The setting of the body rotation is a very important factor along with the setting of the body position, and it can serve as a link that fills in between the motions generated by the movement between holds. The body inclination value calculates the angle value between the body and the hold to be moved according to the current body position and moving direction, converts it into a relative value according to the allowable value of the body inclination, and applies it to change the inclination of the body, similar to the actual get one action.

<head movement>

In sports, 80% of all bodily action inducements rely on vision to acquire information. Because of this, the direction and position of the gaze or head has a great influence on the naturalness of the motion. Based on the effect of the head movement, that is, the gaze on the basic movement concept of climbing, the results on the order and direction can be obtained. If a lookup function for IK control is used, motions can be created by controlling the upper part of the virtual character, including the head and shoulders, to improve the degree of completeness of motions when performing procedural motions.

<Basic operation procedure of climbing>

The procedure for the basic movement of actual climbing begins with the basic posture of making an isosceles triangle by gathering both hands in one hold in a half-standing state after spreading both feet, and repeats steps 1 to 4 to be described later. In the first step, the hand in the direction of the next hold is moved from the basic posture to the next hold. In the second step, the foot opposite to the moved hand is moved to the hold at the position of the opposite foot, and the two feet are brought together to assume an inverted triangle posture. In the third step, move the non-moving foot to an appropriate hold so that an isosceles triangle can be formed by matching the hold position of the hand moved in the first step. In the fourth step, the hand that has not moved is moved to the hold position of the previously moved hand, and both hands are brought together to assume a basic posture.

<Create actions according to procedures>

At this time, when a motion is generated according to the basic motion procedure of actual climbing, an unnatural and artificial motion is generated. This is because, unlike the real human body, the body parts of the virtual character Gak are not controlled associatively according to the movement of Gak. It is possible to newly configure the basic motion procedure in 3D space by performing related control for each part according to the movement direction or position of the virtual character and adding body positioning, tilt setting, and head movement, which are not found in the actual climbing basic motion procedure. The configured procedure is as follows, and the position of the body is reset and interpolated with each step. Climbing may proceed by repeatedly performing the first to sixth steps.

In the first step, the head is interpolated by applying an angle formed between the hold to be moved and the position of the body. In a second step, a corresponding arm part is generated and interpolated to the next posture according to the direction in which the virtual human body is to be moved. In the third step, the inclination of the body is changed according to the position of the moving hold at the same time as the arm is moved and interpolated. In the fourth step, the foot opposite the moving hand is interpolated to the position of the foot in the next posture created. In a fifth step, the existing interpolated and non-moving foot is interpolated to the position of the foot in the next posture. Step 6 interpolates the last non-moving arm to the next generated pose. Along with this, the final body position is interpolated to the next generated posture. Each of these steps can use Unity, a game engine, but is not limited thereto.

When the user terminal 100 scans the identification code, the transmission unit 320 may transmit the stored solution image mapped to the identification code to the user terminal 100 . After scanning an identification code attached to at least one starting point of climbing, the user terminal 100 may output a stored solution image mapped to the identification code. [Identification code-start point-solution image] can be stored in a mapped state, and the user terminal 100 can output it.

The controller 330 may control the user terminal 100 to play the solution video. The AR solution provider 340 identifies the identification code in the image input from the smart glasses via the user terminal 100, and uses the identification code and the color, location, and shape of the photographed rock wall structure as an AR marker. Posture, motion, and route on a rock wall structure can be augmented with AR content and displayed on smart glasses. In this case, the user terminal 100 may interwork with smart glasses. In this case, the solution image may include an augmented reality (AR)-based image that augments an augmented reality (AR)-based posture, motion, and route and outputs the AR content. At this time, the rock wall structure may include an artificial hold.

<Hold Recognition>

Hold recognition technology is important to augment AR content. This is because the recognized information is used in the route registration and mapping system. The hold recognition technology recognizes hold contour information, location information, and rotation information by taking an indoor artificial rock wall image as an input.

<Features of artificial hold>

A hold is a protruding part of a rock that can be held by hand or stepped on by foot. There are as many types of holds as there are different shapes and sizes of rocks. The artificial hold used for indoor artificial rock climbing is used for education and training for natural rock climbing. Various attempts have been made with the artificial hold to improve the tactile feel, grip, and strength. Currently, hold made of polyester and urethane is used to obtain more stable strength. Artificial holds are different in size, color and shape as they pursue diversity. Hold sizes are divided into Mega Holds, 3XL Holds, 2XL Holds, XL Holds, L Holds, M Holds, S Holds, and XS Holds. The color of the hold does not have a separate color. There is no fixed model for the shape of the hold, but there are differences in the grip method according to the specific shape.

The types of holds according to the grip method are Crimp Holds, Sloper Holds, Pocket Holds, Pinch Holds, and Edges Holds. The crimp hold has a hollow upper surface, and the gripped part is about the size of half a finger. Put your fingers together and hold the hollowed place for support. The sloper hold is larger than the palm of your hand and has a convex shape. It has no special protrusions and is slanted so that the fingers do not spread apart and press the fingertips to hold it. A pocket hold is a shape in which a hole is drilled in the hold. Depending on the number of fingers, it is divided into one pocket and two pockets, and you hold it with your fingers straight. The pinch hold is elongated in one direction and the hand grip is bidirectional. The edge hold is held in the form of a long protruding middle part of the hold, as if pinching.

<Hold Recognition Algorithm>

Artificial holds are not uniform in size and shape and have different characteristics. Since there is a problem in which it is difficult to recognize a hold with a certain pattern and rule, it is recognized using the outline information, which is a model of the hold. In order to extract the contour of the artificial hold, a snake algorithm, which is a representative contour-based method, can be used. The snake algorithm extracts the outline of the hold by moving the snake point in the direction that minimizes the energy by the defined energy function. The hold contour may be extracted by moving the initially set snake point to the hold contour using the snake algorithm. A snake point existing outside the hold indicates a snake point initially set to detect the contour. v(s)=(x(s),y(s)) denotes the coordinates of the s-th snake point, and the coordinates move to the outline of the hold due to iterative energy function calculations.

An energy function for moving the initially set snake point to the hold contour is defined as in Equation 1 below.

Here, Eint represents internal energy and Eimage represents external energy. Eimage means the edge of the image calculated by differentiation, and Econ represents the constrained energy defined by the user. The internal energy serves to keep the distance between the snake points constant and smooth the connection. The external energy serves to pull the snake point into the outline of the hold. For the sum of each energy shown in Equation 1, the snake point has the lowest energy value in the hold contour.

<Hold Recognition Process>

In the hold recognition process, hold position information, contour information, and rotation information are recognized by inputting an artificial rock wall image captured by an imaging device. The hold recognition process can be composed of three steps: ① hold contour extraction using snake algorithm, ② contour matching, and ③ recognition information DB registration process. In the first process, the outline is extracted for each hold of the artificial rock wall using the snake algorithm. The hold contour extraction is performed in the management client, and the input image uses the image of the artificial rock wall captured by the imaging device. Contour information is composed of a set of points (x, y). In the second process, the extracted contour information is matched with the reference contour information registered in the root DB of the database. All hold information of the artificial rock wall used in the indoor climbing room is registered in the reference table in advance. Matching is attempted with all registered holds, and scores are measured according to matching similarity, and the hold with the highest score is recognized. The hold can be rotated in any direction when installed on an artificial rock wall, so it rotates in 8 directions and is recognized. Through this process, the outline information, position information, and rotation information of the hold are recognized.

In the third process, the position information, outline information, and rotation information of the recognized hold are registered in the root DB. Recognition information is utilized in route registration and mapping systems. In the route registration system, a route setter selects a hold from an artificial rock wall image to create a route, and at this time, recognition information is used. If the hold selection point is included within the hold contour, the selection is valid, and if not included, the selection is invalid. Similarly, in the route mapping system, recognition information is used when the trainer selects a hold. In addition, in the process of creating an artificial rock wall image in which holds and routes are mapped, recognition information is used to map the selected hold or route to the image.

After extracting the user from the image received from the camera, the posture and motion correction unit 350 measures the user's posture and motion using a skeleton detection algorithm that measures the joint angle of at least one part of the human body. After checking whether the user's posture and motion are similar to the pre-stored posture and motion, a correction request for inducing the pre-stored posture and motion to the user terminal 100 may be transmitted. In this case, the identification code-based climbing solution video service providing system 1 may further include a camera for photographing the user of the user terminal 100 .

The real-time generation unit 360 extracts the user and at least one rock wall structure from the image received from the camera, and generates the next posture and motion in real time according to the posture and motion of the user and the position of the rock structure where the user is located. , may be transmitted to the user terminal 100 so as to be projected on the rock wall structure where the user is located. In this case, when the next posture and motion exceed the preset joint error range, it may be recalculated to regenerate the posture and motion that does not exceed the error range.

At this time, Kinect for posture recognition was originally made to be used for motion recognition games, but there is a method of detecting physical activity using it because it has an excellent price-performance ratio to implement motion recognition functions. Since these methods recognize the shape of an athlete based on the skeleton provided by Kinect and determine the accuracy of the motion, the accuracy of the skeleton information is an important factor in determining the research performance. In particular, the reliability of skeleton information in the method of evaluating motion with joint angles is absolute. However, the skeleton information basically provided by Kinect has the following limitations. Since the skeleton of the Kinect is created based on the shape of the joint part appearing in the depth sensor, the reliability of the skeleton information is reduced due to the instability of the depth sensor information. When a joint part approaches a wall or a large object, the corresponding part is regarded as a part of the wall or large object, and information of the corresponding part of the skeleton becomes unstable. In particular, the limbs that can touch the rock wall can severely distort the skeleton when they touch the floor or wall.

Accordingly, a method of correcting the skeleton can be used. Based on the information provided by Kinect, the skeleton frame stabilization method is used to reduce instability, and a point on the body region where the joint position is detected as the origin is closest to which joint. By discriminating, the body area can be divided into joint parts based on the skeleton joint position. The position of the skeleton joint can be corrected with the midpoint on the joint part divided in this way. The skeleton frame stabilization method sets a high weight to a highly reliable skeleton joint for several frames so that the corresponding joint can contribute more to correction. The criterion for judging the reliability of skeleton information is the tracking status value provided by Kinect and the creation time of the skeleton frame. Tracking status values include “Tracked”, “Inferred”, and “Not Tracked”. “Tracked” indicates that tracking was performed relatively accurately, and “Inferred” indicates that the corresponding body part is closely attached to an object or covered by other body parts or objects. If it is an estimated location, “Not Tracked” means tracking has failed. As for the skeleton creation time, it is determined that the present time is the most reliable, and the more the skeleton is created in the past, the less reliable it is.

Extremity skeleton correction using hold information uses limb region information and hold information. If the detected limb area overlaps with the hold area, the hold position is recognized as the position of the hand and foot, and the center point of the limb area that does not overlap with the hold area is recognized as the position of the hand and foot and corrected. The method of tracking and correcting a skeleton in real time using two Kinects is difficult to accurately obtain skeleton information with one Kinect. A method of extracting a skeleton in a more stable form may be used by using weights according to the reliability of joint position information in the skeleton for each kinect. However, since it is a method of calibration using only the skeleton information provided by Kinect, accuracy is low in a situation where the skeleton information is greatly distorted from the beginning, and additional installation cost and calibration work to align the skeleton coordinate system between devices is required.

<Classification of body regions into joint regions>

This method generates information for each body part required to generate an event for each body part used in the game. Since each body part is created for each joint of the skeleton, it is defined as a joint area in an embodiment of the present invention. In order to classify the climber's body region into joint regions, limb corrected skeleton information is used. The climber's body region and the skeleton information are placed at the same location, and the recognition region for each joint of the skeleton is expanded to classify the climber's body region by body part. The recognition area starts to expand simultaneously at the joint point for each body part in the skeleton, and gradually expands to the same size. When all areas of the climber area are divided by body part, expansion may be completed. The recognition area is extended even when the skeleton joint is out of area.

<Body area based skeleton correction>

<Fix abnormal skeleton joints>

In order to classify body regions into regions around joints, the reliability of skeleton information must be high. Since the skeleton obtained by the general skeleton frame stabilization method uses only the above tracking state to determine the reliability of the skeleton information, it is determined that the skeleton is highly reliable because the skeleton tracking state is “Tracked” even though the skeleton is actually severely broken. Mis-correction happens. Therefore, in one embodiment of the present invention, a method of additionally correcting abnormal skeleton joint positions may be used to compensate for this. In most cases, the skeleton information comes out stably, but depending on the situation, it may be severely distorted or twisted. When such distorted skeleton information is obtained, the reliability of the skeleton information deteriorates even when the above-described skeleton correction method is used. Therefore, the probability that incorrect skeleton information is reflected is minimized by excluding the incorrect skeleton information and increasing the weight of the correct skeleton information acquired at the previous time.

The criterion for detecting erroneous skeleton information is the position change of joints per frame. Male speed climbers, who are the fastest climbers, take about 6 seconds to climb a 15m high wall, which averages just over 2.5m/s. In the Kinect device used to implement the method according to an embodiment of the present invention, since skeleton information can be obtained 30 times per second, a moving speed of 2.5 m per second is about 8.3 cm per frame. Therefore, in one embodiment of the present invention, the movement distance of a specific joint between frames may be set to 10 cm (equivalent to 3 m per second) with a slight margin in the climbing speed.

The process of correcting abnormal skeleton joints is as follows. First, the movement distance for each joint is obtained by comparing it with the previous frame. At this time, if the movement distance exceeds a certain limit (10cm per frame), it is judged to be incorrect and the tracking state of the corresponding joint is redefined as “Inferred”. After that, the skeleton frame stabilization algorithm is performed.

<Finding the point of focus for each joint>

The classified joint region is used for body region-based skeleton correction. The center of gravity is usually used as the midpoint representing each joint part. As a general method for obtaining the center of gravity, there is a method for obtaining the center of gravity of a polygon, but it is not suitable for joints according to an embodiment of the present invention. This is because, in order to use the method of obtaining the center of gravity of a polygon, additional work is required for polygon approximation after the outline extraction operation for each joint part. This is because it is likely to consist of a dog's mass. Therefore, the joint area is subdivided into pixels having the same size, and since the formula for the center of gravity used in this case is the same as the formula for obtaining the center of mass, a detailed description thereof will be omitted. The center of mass is the central point of the mass of the entire object, and acts with the external system as if the total mass were at the center of mass. If the gravity is uniform, it is the same as the center of gravity, so it is sometimes used interchangeably. At this time, the point of action of the sum of the gravitational forces acting on each part of the object is called the center of gravity.

Here, each pixel is a piece of the same size that makes up a massive object, and each piece has the same mass because each piece has the same density. Therefore, we use a formula that gives the same mass for all pixel positions and averages them.

<Skeleton correction using joints>

Although the above-described skeleton frame stabilization method can solve the instability of Kinect skeleton information to some extent, it is not sufficient for recognizing a climber's posture. In order to compensate for this and obtain skeleton information with higher reliability, a skeleton correction method using joint parts is used. For example, it is abnormal if some joint positions protrude from the body region when compared with the body region. Since it is normal for all joint parts to be part of the body area, it is necessary to correct the position of the protruding joint so that it is within the body area. To this end, a reference position is required when each joint position is corrected, and the center of gravity obtained for each joint part described above can be used. In this way, any joint is located within the body region, so that even if the skeleton is broken, the severity can be alleviated. If there is no joint part associated with a specific joint, the results from the skeleton frame stabilization step are used. The skeleton joint corrected in this way is regarded as correct information, and the tracking state can be updated to “Tracked”. Of course, various methods other than the above method may be used.

When at least one climbing image is uploaded from the user terminal 100, the solution providing unit 370 compares the location information of the at least one climbing image with the location information of the pre-stored solution image, and analyzes the captured image. A pre-stored solution image having location information and location information within a preset radius may be provided to the user terminal 100 .

The deep learning analysis unit 380 maps and stores the location, color, and shape of at least one rock structure to an identification code, maps and stores at least one solution image when located on at least one rock structure, and When at least one rock structure is input from the terminal 100, after identifying the rock structure using a deep learning-based algorithm, a pre-mapped and stored solution image may be provided. In the above-described method, the hold, which is a rock wall structure, was used by the snake algorithm, but in deep learning, the position, color, and shape of each hold can be combined and read based on an image. When using location information, information of the Z axis is required in addition to XY, which can be solved by using information written on the height on an artificial rock wall or a mark attached to each artificial hold. Once the location is resolved, color and shape can use deep learning-based YOLO. Alternatively, a rock wall corresponding to the problem can be photographed from a distance using a detection or classification model, which is a deep learning technology, and then compared with representative photos of the problem to specify what kind of problem it is. For this, various deep learning, for example, CNN may be used, but is not limited thereto.

Hereinafter, an operation process according to the configuration of the above-described video service providing server of FIG. 2 will be described in detail with reference to FIGS. 3 and 4 as examples. However, it will be apparent that the embodiment is only any one of various embodiments of the present invention, and is not limited thereto.

Referring to FIG. 3, (a) when the user terminal 100 scans an identification code printed at the start point and attached or attached in the form of a sticker, the user terminal 100 is connected to a URL pre-mapped to the identification code, The pre-stored solution image is output. At this time, when the user terminal 100 wears smart glasses, for example, Google Glass, the image input via the user terminal 100 is analyzed by the video service providing server 300, and the user is currently located. , and then AR content can be extracted from the solution image and transmitted to the smart glasses via the user terminal 100 to guide the motion, posture, and route. In this case, the user terminal 100 can intuitively know which direction, which posture, and which motion to take even in a state of holding the hold. As shown in (a) of FIG. 4 , when the wearable device 400 is interlocked with the user terminal 100, the video service providing server 300 uses the data collected from the wearable device 400 to provide a corresponding form and format. If the solution image of can be transmitted to the wearable device 400 via the user terminal 100, and the user's entire appearance can be captured with a camera as in (b), as described above, use Kinect or the like After accurately extracting the user's skeleton information, the next posture and motion may be extracted and guidance and instructions may be given to the user.

Matters not described for the identification code-based climbing solution video service providing method of FIGS. 2 to 4 are the same as those described for the identification code-based climbing solution video service providing method through FIG. Since it can be easily inferred, the following description will be omitted.

FIG. 5 is a diagram illustrating a process of transmitting/receiving data between components included in the identification code-based climbing solution video service providing system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described through FIG. 5, but the present application is not limited to such an embodiment, and according to various embodiments described above, It is obvious to those skilled in the art that a process of transmitting and receiving data may be changed.

Referring to FIG. 5, the video service providing server maps and stores the coordinates of at least one starting point, at least one identification code, and a solution image guiding the attitude, motion, and route to be taken at the at least one starting point ( S5100), when the user terminal scans the identification code, the solution image previously mapped to the identification code and stored is transmitted to the user terminal (S5200).

And, the video service providing server controls the user terminal to play the solution video (S5300).

The order between the above-described steps (S5100 to S5300) is only an example, and is not limited thereto. That is, the order of the above-described steps (S5100 to S5300) may be mutually changed, and some of the steps may be simultaneously executed or deleted.

Matters that have not been explained in the identification code-based climbing solution video service provision method of FIG. 5 are the same as those described for the identification code-based climbing solution video service provision method through FIGS. 1 to 4 above, or from the description. Since it can be easily inferred, the following description will be omitted.

The identification code-based climbing solution video service providing method according to an embodiment described with reference to FIG. 5 may be implemented in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. can Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In the above-described identification code-based climbing solution video service providing method according to an embodiment of the present invention, by an application basically installed in a terminal (this may include a program included in a platform or operating system basically loaded in the terminal) It may be executed, or it may be executed by an application (ie, a program) directly installed in the master terminal by a user through an application providing server such as an application store server, an application or a web server related to the corresponding service. In this sense, the above-described identification code-based climbing solution video service providing method according to an embodiment of the present invention is implemented as an application (ie, a program) that is basically installed in a terminal or directly installed by a user, and is implemented as a computer such as a terminal. It can be recorded on a readable recording medium.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (10)

After scanning an identification code attached to at least one starting point of climbing, a user terminal that outputs a stored solution image mapped to the identification code; and
A database unit for mapping and storing the coordinates of at least one starting point, at least one identification code, and a solution image guiding the posture, motion, and route to be taken at the at least one starting point, and scanning the identification code at the user terminal. a video service providing server including a transmission unit that transmits the stored solution video mapped to the identification code to the user terminal, and a control unit that controls the user terminal to play the solution video;
Identification code-based climbing solution video service providing system comprising a.
According to claim 1,
The solution video is an identification code-based climbing solution video service providing system, characterized in that it includes an AR-based video that augments AR (Augmented Reality)-based posture, motion and route and outputs it as AR content.
According to claim 1,
The user terminal is interlocked with smart glasses,
The video service providing server,
Identify the identification code in the image input from the smart glasses via the user terminal, and use the color, position, and shape of the rock structure photographed together with the identification code as an AR marker to determine the posture and motion on the rock structure. and an AR solution providing unit that augments the route with AR content and displays it on the smart glasses.
Identification code-based climbing solution video service providing system further comprising a.
According to claim 1,
The identification code-based climbing solution video service providing system,
a camera for photographing the user of the user terminal;
Identification code-based climbing solution video service providing system further comprising a.
According to claim 4,
The video service providing server,
After extracting the user from the image received from the camera, the user's posture and motion are measured using a skeleton detection algorithm that measures the joint angle of at least one part of the body, and the user's posture and motion a posture/motion correcting unit for transmitting a correction request for inducing the pre-stored posture and motion to the user terminal after checking whether the motion is similar to the pre-stored posture and motion;
Identification code-based climbing solution video service providing system further comprising a.
According to claim 4,
The video service providing server,
After extracting the user and at least one rock wall structure from the image received from the camera, generating the next posture and motion in real time according to the posture and motion of the user and the position of the rock structure where the user is located, a real-time generating unit for transmitting to the user terminal to be projected onto the rock wall structure;
Identification code-based climbing solution video service providing system further comprising a.
According to claim 6,
When the next posture and motion exceed a preset joint error range, it is recalculated to regenerate the posture and motion that does not exceed the error range.
According to claim 3,
The rock wall structure is an identification code-based climbing solution video service providing system, characterized in that it comprises an artificial hold (Hold).
According to claim 1,
The video service providing server,
When at least one climbing image is uploaded from the user terminal, location information of the at least one climbing image is compared with location information of a pre-stored solution image for analysis, A solution providing unit for providing a pre-stored solution image having location information to the user terminal;
Identification code-based climbing solution video service providing system further comprising a.
According to claim 1,
The video service providing server,
The location, color, and shape of at least one rock wall structure are mapped to an identification code and stored, and at least one solution image when located on the at least one rock wall structure is mapped and stored, and the at least one rock wall structure is stored from the user terminal. When a structure is input, a deep learning analysis unit that identifies a rock wall structure using a deep learning-based algorithm and then provides a pre-mapped and stored solution image;
Identification code-based climbing solution video service providing system further comprising a.

Images