KR20210031333A - Augmented reality training system - Google Patents

Abstract

The present invention relates to an augmented reality training system which enables a participant to easily learn based on preset training information and to check the participant′s training state by representing the participant′s training state in augmented reality. To this end, the augmented reality training system comprises: an experience information collection unit which collects motion information of a participant; a participant management unit which generates shape data by shaping the motion information of a participant in three dimensions; a training management unit which includes at least one of a learning state analysis unit that generates state data by comparing and analyzing shape data and preset training information and a learning progress analysis unit that generates progress data by comparing and analyzing current shape data and previous shape data in correspondence with the preset training information; and a training server in which the participant information for the identification of the participant, the motion information of a participant, preset training information, shape data, and analysis data are stored.

Description

Augmented Reality Training System {AUGMENTED REALITY TRAINING SYSTEM}

The present invention relates to an augmented reality training system, and more specifically, an augmented reality that allows an experienced person to learn easily based on preset training information, and expresses the experiencer's training status in augmented reality to confirm the experiencer's training status. It relates to a real training system.

In general, augmented reality (Augmented Reality) is a term derived from a virtual environment and virtual reality, refers to the addition of information generated by a computer on the real world. That is, in the real world information, there may be information that the user does not need, and sometimes the information that the user needs may be insufficient. However, if you use a computer-generated virtual environment, you can simplify information you do not need or create invisible information. That is, the augmented reality system combines the real world with the virtual world to allow interaction with the user in real time, and has the advantage of providing a more improved sense of immersion than the virtual reality system.

In addition, modern people are increasing the amount of time spent on indoor activities due to fine dust outdoors and inappropriate weather outdoors, and problems such as obesity and depression have occurred due to decreased activity. Accordingly, in recent years, there is an increasing demand for systems capable of indoors for training such as golf, baseball, and choreography.

Republic of Korea Patent Publication No. 10-1950243 (Name of invention: Golf practice device providing augmented reality or virtual reality experience, 2019. 02. 21. Announcement)

An object of the present invention is to solve the problems of the prior art, and an augmented reality that allows an experienced person to easily learn based on preset training information, and to check the training status of an experienced person by expressing the experience person's training status in augmented reality. It is to provide a training system.

According to a preferred embodiment for achieving the object of the present invention described above, the augmented reality training system according to the present invention includes an experience information collection unit that collects motion information of an experienced person; An experience user management unit for generating shape data by shaping the motion information of the experienced person in three dimensions; A learning state analysis unit that compares and analyzes the shape data and preset training information to generate state data, and a learning progress that generates progress data by comparing and analyzing the current shape data and previous shape data in response to the preset training information. A training management unit for generating analysis data including at least one of the analysis units; And a training server in which the experience user information for identification of the experience person, motion information of the experience person, preset training information, the shape data, and the analysis data are stored.

Here, the experience information collection unit includes: a camera unit that photographs an experienced person at three or more points spaced apart from each other based on the experience person and generates photographing data for the experience person at each point; And a lidar unit that generates lidar data for motion information of the experiencer by extracting reflection of light (laser) output to the experiencer by matching with preset location information for the camera unit.

Here, the experience user management unit includes: an image calculation unit for forming a first superimposed image by superimposing photographing data on an experience person for each point in a three-dimensional space; An image adjustment unit for primary correction of the first superimposed image using a depth algorithm relative to the three-dimensional space from the photographing data of the experienced person at each point; A lidar calculation unit for forming a second superimposed image by superimposing lidar data on the experienced person at each point in a three-dimensional space; And a three-dimensional conversion unit that combines the first superimposed image and the second superimposed image corrected through the image adjustment unit and converts the first superimposed image into shape data for a three-dimensional space.

The augmented reality training system according to the present invention is matched with the experience user information and communicated with the training server by wire, and a training display expressing at least preset training information among preset training information, the shape data, and the analysis data as an image unit; And a mobile terminal that is matched with the experience user information and wirelessly communicates with the training server, and expresses at least the shape data of preset training information, the shape data, and the analysis data as an image in augmented reality. It further includes at least any one of.

Here, the experience user management unit further includes a content generation unit that provides the shape data as content that can be rotated 360 degrees in augmented reality, wherein the mobile terminal expresses the content.

Here, the training management unit includes: a calibration selection unit for selecting a portion to be corrected from preset training information or the shape data in response to an analysis result of at least one of the learning state analysis unit and the learning progress analysis unit; Include more.

Here, the training management unit further includes an image duplication unit that duplicates at least one of the preset training information and the shape data in two or more.

Here, the experience information collecting unit further includes a haptic glove worn on the hand of the experiencer to sense the movement of the hand of the experiencer to generate haptic data, wherein the three-dimensional conversion unit comprises: a first corrected by the image adjusting unit When the superimposed image and the second superimposed image are combined, the haptic data is further combined to supplement the shape data for a three-dimensional space.

Here, the experience information collection unit includes: a plurality of infrared markers coupled to the body of the experiencer; And an infrared camera that is matched with preset position information for the camera unit to extract the position of the infrared marker to generate marker data, wherein the marker data is the first corrected by the image adjustment unit. When the first superimposed image is secondarily corrected by combining with the superimposed image, or when the first superimposed image and the second superimposed image corrected through the image adjustment unit are combined in the stereoscopic conversion unit, the three-dimensional space is further combined. The shape data for is supplemented.

Here, the experience information collection unit further includes a reference sticker attached to a specific part of the experienced person in response to preset training information, and the training management unit further includes the reference sticker and reference information included in the preset training information It further includes; experience matching unit for matching.

Here, the training management unit further includes an experienced person matching unit for matching the basic posture of the experienced person included in the shape data with the training posture included in the preset training information.

According to the augmented reality training system according to the present invention, the experiencer can easily learn based on preset training information, and the training status of the experiencer can be confirmed by expressing the training status of the experiencer in augmented reality.

In addition, according to the present invention, the movement of the experiencer can be obtained as it is through the camera unit, the depth information on the movement of the experiencer can be obtained through the lidar unit, and the experiencer from the shape data through the detailed configuration of the experiencer management unit. You can express a three-dimensional effect according to the movement of

In addition, according to the present invention, it is possible to conveniently check preset training information for learning of the experiencer and the training state of the experiencer through at least one of the training display unit and the portable terminal.

In addition, the present invention can check the training status of the experiencer from various angles through the content generation unit.

In addition, the present invention is to check in more detail the areas requiring correction (eg, head, torso, arms, inclination of legs, rotation of the waist, position of the hips, etc.) in response to the analysis result of the training management unit through the correction selection unit. You can check the training status of the experienced person in detail for the area that needs correction.

In addition, the present invention can express two or more preset training information and shape data in augmented reality through the image replicating unit, and when the preset training information corresponds to the choreography, it is possible to reproduce the group dance or group image, and Depending on the adjustment status, it can be reproduced as if it were a video shot by many people, not just copying.

In addition, according to the present invention, when the training information preset through the haptic glove participates in learning by gripping the tool, the experiencer can check the gripping state of the tool in detail.

In addition, according to the present invention, photographing data may be additionally corrected through an infrared marker and an infrared camera, or accuracy of shape data may be improved.

In addition, the present invention can more accurately match the direction of the shape data and the direction of preset training information through the experience person matching unit, and improve the reliability of the analysis result of the training management unit.

1 is a plan view showing an augmented reality training system according to an embodiment of the present invention.
2 is a block diagram showing an augmented reality training system according to an embodiment of the present invention.
3 is a plan view showing a haptic glove in an augmented reality training system according to an embodiment of the present invention.

Hereinafter, an embodiment of an augmented reality training system according to the present invention will be described with reference to the accompanying drawings. At this time, the present invention is not limited or limited by the examples. In addition, in describing the present invention, detailed descriptions of known functions or configurations may be omitted to clarify the gist of the present invention.

1 to 3, the augmented reality training system according to an embodiment of the present invention includes an experience information collection unit 10, an experience person management unit 20, a training management unit 30, and a training server ( 50), and may further include at least one of the training display unit 40 and the portable terminal 60.

Here, the user interface may include one or more input devices such as a touch screen, a keypad, a keyboard, a stylus, and other input devices. For example, the user interface may include a touch screen provided in the mobile terminal 60.

The experience information collection unit 10 collects motion information of the experience person T.

Then, when the start signal is input to the user interface, the training display unit 40 is operated so that the preset training information is expressed visually and aurally, and the experience information collecting unit 10 is operated to be located in the experience area (S). It is possible to collect motion information of the experienced person (T). At this time, if the experienced person T is not searched in the experience area S, an alarm may be generated.

The experience information collection unit 10 is a camera unit 11 that photographs the experienced person T at three or more points separated from each other based on the experienced person T, and generates photographing data for the experienced person T at each point. And, a lidar unit that extracts the reflection of light (laser) that is matched with preset location information for the camera unit 11 and output to the experienced person (T) to generate lidar data for the motion information of the experienced person (T). It may include.

In one embodiment of the present invention, the camera unit 11 and the lidar unit 12 are respectively spaced apart from each other along the circumference of the experience area S based on the experience area S, and are arranged at the vertices of the virtual triangle. Although the structure is shown, the number and location of the camera units 11 are not limited, and the experiencer (T) moving in the experience area (S) through various known forms such as a vertex of a virtual polygon and a vertex of a virtual tetrahedron. Can be taken stably.

The experience information collection unit 10 is matched with a plurality of infrared markers 13 coupled to the body of the experienced person T and preset location information for the camera unit 11 to extract the location of the infrared marker 13 An infrared camera 14 for generating marker data may be further included.

The experience information collection unit 10 may further include a haptic glove 15 that is worn on the hand of the experience person T and senses the movement of the hand of the experience person T to generate haptic data. The haptic glove 15 includes at least one of a bending sensor 151 that detects the movement of the hand of the experienced person T and a contact sensor 152 that detects the contact state of a tool that is in contact with the hand of the experience person T. It can be provided.

The experience information collection unit 10 may further include a reference sticker (not shown) attached to a specific portion of the experience person T in response to preset training information. As an example, the reference sticker (not shown) may include a target portion attached to the chest of the experienced person T.

The experience person management unit 20 generates shape data by forming the motion information of the experience person T in three dimensions.

The experience user management unit 20 includes an image calculation unit 21 that forms a first superimposed image by superimposing photographing data on the experience person T at each point in a three-dimensional space, and a photographing of the experience person T at each point. The image adjustment unit 22 for primary correction of the first superimposed image using a depth algorithm relative to the 3D space from the data, and the lidar data for the experienced person T for each arbitrary point are superimposed in the 3D space to obtain a second A lidar calculation unit 23 for forming an overlapping image, and a three-dimensional conversion unit 24 that combines the first superimposed image and the second superimposed image corrected through the image adjustment unit 22 and converts it into shape data for a three-dimensional space. ) Can be included.

Then, the photographing data and the lidar data are transmitted to the image calculation unit 21 and the lidar calculation unit 23 in a first-in, first-out manner according to the time series order, and the image calculation unit 21, the image adjustment unit 22 and the lidar. It is converted into shape data through the calculation unit 23 and the three-dimensional conversion unit 24. Here, the preset training information may be divided according to a time series order by unit frame and unit time and expressed continuously, and correspondingly, the shape data may be divided according to a time series order by unit frame and unit time.

When the experience information collection unit 10 includes a haptic glove 15, the three-dimensional conversion unit 24 combines the first superimposed image and the second superimposed image corrected through the image adjustment unit 22, haptic data The shape data for the three-dimensional space can be supplemented by additionally combining.

When the experience information collection unit 10 includes an infrared camera 14, the image adjustment unit 22 combines the first-corrected first superimposed image and marker data to correct the first superimposed image, or perform a three-dimensional transformation. When combining the first superimposed image and the second superimposed image corrected through the image adjustment unit, the unit 24 may supplement shape data for the 3D space by additionally combining the marker data.

In other words, when the experience information collection unit 10 includes the infrared camera 14, the marker data is combined with the first superimposed image that is first corrected by the image adjustment unit 22 to perform secondary correction of the first superimposed image. Alternatively, when the 3D conversion unit 24 combines the first superimposed image and the second superimposed image corrected through the image adjustment unit 22, the shape data for the 3D space may be supplemented by additionally combining the first superimposed image and the second superimposed image.

The experience user management unit 20 may further include a content generation unit 25 that provides shape data as content that can be rotated 360 degrees in augmented reality.

Then, when a content reproduction signal is input to the user interface, the content stored in the content storage unit 56 of the training server 50 is transmitted to the mobile terminal 60, and the mobile terminal 60 visually displays the content in augmented reality. Can be expressed aurally.

The training management unit 30 compares and analyzes the shape data to generate analysis data.

The training management unit 30 compares the current shape data with the previous shape data in response to the learning status analysis unit 32 that generates status data by comparing and analyzing the shape data and preset training information. It may include at least one of the learning progress analysis unit 33 that analyzes and generates progress data.

Accordingly, the analysis data may include at least one of state data and shape data.

The training management unit 30 is a part (for example, a head, a part that needs to be corrected in the preset training information or shape data in response to the analysis result of at least one of the learning state analysis unit 32 and the learning progress analysis unit 33). It may further include a correction selection unit 34 for selecting the inclination of the torso, arms and legs, the rotational state of the waist, the position of the hips, etc.).

Then, the user interface may input a position that requires correction from preset training information expressed on the training display unit 40 or shape data expressed on the mobile terminal 60, and the correction selection unit 34 Including, it can be selected as a part that needs correction (for example, the tilt of the head, torso, arms and legs, the rotation of the waist, the position of the hips, etc.)

The training management unit 30 may further include an image duplicating unit 35 that duplicates at least one of preset training information and shape data in two or more.

Then, when a duplicate signal is input to the user interface, the image duplicating unit 35 may duplicate at least one of preset training information and shape data in two or more.

For example, when the experience information collection unit 10 includes a reference sticker (not shown), the training management unit 30 matches the reference sticker (not shown) with reference information included in preset training information. It may further include a portion (31).

As another example, the training management unit 30 may further include an experienced person matching unit 31 for matching the basic posture of the experienced person T included in the shape data with the training posture included in the preset training information. Here, the basic posture of the experienced person T may be collected by the experience information collecting unit 10 as part of the motion information of the experienced person T according to the training posture included in the training information preset in the initial experience area S. .

The training server 50 stores experience user information for identification of the experience person T, motion information of the experience person T, preset training information, shape data, and analysis data. The data stored in the training server 50 may be updated in response to the motion information of the experienced person T.

The training server 50 includes an experience user information storage unit 51 that stores experience user information, a training information storage unit 52 that stores preset training information, and photographing data and haptic gloves 15 among motion information of the experience user (T). An image storage unit in which the haptic data generated by the infrared camera 14, the marker data generated by the infrared camera 14, the first superimposed image generated by the image calculation unit 21, and the corrected image generated by the image adjustment unit 22 are matched and stored A lidar storage unit 54 in which the lidar data generated by the lidar unit 12 and the second superimposed image generated by the lidar calculation unit 23 are matched and stored, and a three-dimensional conversion unit The three-dimensional storage unit 55 that stores the shape data generated in (24), the content storage unit 56 that stores the content generated by the content generation unit 25, and the learning state analysis unit 32 An analysis storage unit 57 for storing state data and progress data generated by the learning progress analysis unit 33 may be included.

In addition, the training server 50 may further include a terminal communication unit 58 for wireless communication with the portable terminal 60.

The experience user information may be extracted from terminal information of the mobile terminal 60 through wireless communication with the mobile terminal 60, or may be input through a user interface.

The training display unit 40 is matched with the experience user information and communicates with the training server 50 by wire or wirelessly, and displays at least preset training information among preset training information, shape data, and analysis data as an image. The training display unit 40 may express at least preset training information as an image according to a start signal input to a user interface or a separate display signal.

Here, the training display unit 40 is not limited, and may represent various known forms such as a hologram and a display.

The mobile terminal 60 is matched with the experience user information and wirelessly communicates with the training server 50, and displays at least shape data among preset training information, shape data, and analysis data in augmented reality as an image.

When the experience user management unit 20 includes the content generation unit 25, the mobile terminal 60 may express the content provided by the content generation unit 25 according to the content reproduction signal.

A training app that works with the training server 50 is installed on the mobile terminal 60.

According to the augmented reality training system described above, the experienced person T can easily learn based on preset training information, and the training status of the experienced person T can be checked by expressing the training status of the experienced person T in augmented reality. have.

In addition, through the camera unit 11, the movement of the experienced person T can be obtained as it is, and the depth information on the movement of the experienced person T can be obtained through the lidar unit 12, and the experienced person management unit Through the detailed configuration of (20), it is possible to express a three-dimensional effect according to the movement of the experienced person (T) in the shape data.

In addition, preset training information for learning of the experienced person T and the training state of the experienced person T can be easily checked through at least one of the training display unit 40 and the portable terminal 60.

In addition, through the content generation unit 25, the training status of the experienced person T can be checked from various angles.

In addition, the area that needs correction in response to the analysis result of the training management unit 30 through the correction selection unit 34 (for example, the head, the torso, the inclination of the arms and legs, the rotation of the waist, the position of the hips, etc.) You can check in more detail, and you can check the training status of the experienced person (T) in detail for the area that needs correction.

In addition, it is possible to express two or more preset training information and shape data in augmented reality through the image duplicating unit 35, and if the preset training information corresponds to the choreography, it is possible to reproduce the group dance or group image, and the image reproduction Depending on the control state of the unit 35, it is possible to reproduce not only a simple copy, but as if it is an image captured by a large number of people.

In addition, when training information preset through the haptic glove 15 allows the experienced person T to grasp the tool and participate in learning, the experiencer T can check the gripping state of the tool in detail.

In addition, the photographing data may be additionally corrected through the infrared marker 13 and the infrared camera 14 or the accuracy of the shape data may be improved.

In addition, it is possible to more accurately match the direction of the shape data and the direction of preset training information through the experience person matching unit 31 and improve the reliability of the analysis result of the training management unit 30.

As described above, preferred embodiments of the present invention have been described with reference to the drawings, but those skilled in the art will variously modify the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. Can be modified or changed.

10: experience information collection unit 11: camera unit 12: lidar unit
13: infrared marker 14: infrared camera 15: haptic gloves
151: bending sensor 152: contact sensor 20: experience user management unit
21: image calculation unit 22: image adjustment unit 23: lidar calculation unit
24: three-dimensional conversion unit 25: content generation unit 30: training management unit
31: experience matching unit 32: learning state analysis unit 33: learning progress analysis unit
34: correction selection unit 35: image reproduction unit 40: training display unit
50: training server 51: experience user information storage unit 52: training information storage unit
53: image storage unit 54: lidar storage unit 55: stereoscopic storage unit
56: content storage unit 57: analysis storage unit 58: terminal communication unit
60: mobile terminal T: experienced person S: experience area

Claims (6)

An experience information collection unit that collects motion information of an experienced person;
An experience user management unit for generating shape data by shaping the motion information of the experienced person in three dimensions;
A learning state analysis unit that compares and analyzes the shape data and preset training information to generate state data, and a learning progress that generates progress data by comparing and analyzing the current shape data and previous shape data in response to the preset training information. A training management unit for generating analysis data including at least one of the analysis units; And
Including; a training server for storing the experience person information for identification of the experience person, the experience person's motion information, preset training information, the shape data, and the analysis data,
The experience information collection unit,
A camera unit that photographs the experienced person at three or more points separated from each other based on the experienced person, and generates photographing data for the experienced person at each point; And
Including; a lidar unit for generating lidar data on the motion information of the experiencer by extracting the reflection of light (laser) output to the experiencer by matching with preset location information of the camera unit,
The experience user management unit,
An image calculation unit for forming a first superimposed image by superimposing photographing data on an experienced person at each point in a three-dimensional space;
An image adjustment unit for primary correction of the first superimposed image using a depth algorithm relative to the three-dimensional space from the photographing data of the experienced person at each point;
A lidar calculation unit for forming a second superimposed image by superimposing lidar data on the experienced person at each point in a three-dimensional space; And
And a three-dimensional conversion unit that combines the first superimposed image and the second superimposed image corrected through the image adjustment unit and converts it into shape data for a three-dimensional space.
The method of claim 1,
A training display unit that matches the experience user information and communicates with the training server by wire, and expresses at least preset training information from among preset training information, the shape data, and the analysis data as an image; And
A mobile terminal that is matched with the experience user information and wirelessly communicates with the training server, and expresses at least the shape data of preset training information, the shape data, and the analysis data as an image in augmented reality;
Augmented reality training system, characterized in that it further comprises at least any one of.
The method of claim 2,
The experience user management unit further includes; a content generation unit that provides the shape data as content that can rotate 360 degrees in augmented reality,
The mobile terminal,
Augmented reality training system, characterized in that expressing the content.
The method of claim 2,
The training management unit may further include a calibration selection unit for selecting a portion to be corrected from preset training information or the shape data in response to an analysis result of at least one of the learning state analysis unit and the learning progress analysis unit. Augmented reality training system, characterized in that to.
The method of claim 2,
The training management unit further comprises an image duplicating unit that duplicates at least one of preset training information and the shape data in two or more.
The method according to any one of claims 1 to 5,
The experience information collection unit further includes a haptic glove worn on the hand of the experiencer to sense the movement of the hand of the experiencer to generate haptic data, and
The three-dimensional conversion unit,
When combining the first superimposed image and the second superimposed image corrected through the image adjustment unit, the haptic data is additionally combined to supplement the shape data for a three-dimensional space.

Images