KR20220039973A - Educational system based augmented reality and control method thereof - Google Patents

Abstract

The present invention relates to an augmented reality-based training system, which includes: a trainee terminal capable of photographing training equipment and outputting training information; an administrator terminal setting an object for training and managing a training situation; an object generator setting an object for operation and maintenance training with respect to the training equipment; and a training server extracting an object based on video information and motion information input from the trainee terminal, extracting training information matching the object from a training information DB, performing augmented reality-based output to the trainee terminal, performing training by providing training object-specific training information from the administrator terminal to a selected trainee terminal, managing the training situation, setting the object input from the object generator, and storing and managing the object in the training information DB.

Description

Augmented reality-based education system {EDUCATIONAL SYSTEM BASED AUGMENTED REALITY AND CONTROL METHOD THEREOF}

The present invention relates to an augmented reality-based education system, and more particularly, augmented reality that enables operation and maintenance training of equipment while shooting control equipment based on augmented reality through a head mounted display or a tablet PC. It is about an educational system based on

In general, a head mounted display (HMD) refers to various digital devices that are worn on the head like glasses to receive multimedia content. According to the trend of weight reduction and miniaturization of digital devices, various wearable computers are being developed, and HMDs are also widely used. HMD can provide various conveniences to users by combining with augmented reality technology and N-screen technology beyond a simple display function.

A user wearing the HMD can interact with the HMD in various situations. In this case, in order for the HMD to provide a better context-aware service to users, a simpler user interface is needed.

In addition, efforts are being made to apply an Augmented Reality (AR) service that displays information by superimposing information on an actual image using an HMD.

Most of these augmented reality services use a smartphone, and when an image of an area that a user wants to be augmented is acquired through a camera, specific information is superimposed inside/outside it using the feature points of the object, or, more realistically, a device After confirming the location using the GPS, gyro, and altitude sensor installed in the device, the location information is superimposed on the surrounding information using the distance information measured indirectly through the camera using the location as the starting point.

The background technology of the present invention is disclosed in Korean Patent Application Laid-Open No. 10-2016-0009213 (published on January 26, 2016, a head mounted display and a content providing method using the same).

Recently, a simulation system for training of the distribution system has been built and used to increase the skill level of the operator for handling failure events.

That is, the training center is operated for the purpose of training to cope with the failure occurring on the basis of the measurement information (One Snap-Shot) measured on an arbitrary line.

The current distribution system operating system performs a function of remotely controlling and monitoring automation equipment to monitor failures occurring in the distribution system and perform line work.

The training simulation system for training the use of the current operating system generates an event (failure) and is configured in the form of training only the operator's ability to operate an automated facility to handle it.

It is necessary to improve proficiency by frequently training through operation and maintenance training for equipment while checking the control system directly in the field based on augmented reality by convergence with such a virtual simulation system.

An object of the present invention according to one aspect is to provide an augmented reality-based education system that enables operation and maintenance training for equipment while shooting control equipment based on augmented reality through a head mounted display or a tablet PC .

An augmented reality-based education system according to an aspect of the present invention includes: a trainee terminal capable of photographing educational equipment and outputting educational information; an administrator terminal for setting an object for education and managing the educational situation; Object generator for setting objects for operation and maintenance training for educational equipment; And the object is extracted based on the image information and motion information input from the trainee terminal, the education information matching the object is extracted from the education information DB, and output to the trainee terminal based on augmented reality, and the education object is sent to the trainee terminal selected from the manager terminal It is characterized in that it comprises a; education server that provides education information according to the education, manages the education situation, sets the object input from the object generator and stores and manages the education information DB.

In the present invention, the trainee terminal is characterized in that it includes an operation program for performing operation and maintenance education of educational equipment by accessing the education server based on UI (User Interface)/UX (User Experience).

In the present invention, the trainee terminal is characterized in that it displays educational information for operation and maintenance education of educational equipment in three dimensions around an object, and displays the education information of the corresponding position according to the moving direction of the screen.

In the present invention, the manager terminal is characterized in that it includes an operation control program for managing trainees, conducting and evaluating maintenance training, managing network status, and monitoring the training progress.

In the present invention, the object generator includes: an object management unit for managing component information used in operation and maintenance training for educational equipment; an effect management unit that expresses a reaction method when a component is manipulated; a step management unit that manages the educational procedure scenario; Education management department that manages education progress result information; and a network connection unit for managing a connection with the education server and transmitting data.

In the present invention, the education server provides information output to the trainee terminal in a cybersecurity environment to the manager terminal.

The education server in the present invention is characterized in that it is built based on a REST (Representational State Transfer)ful framework.

In the present invention, the education server includes: a simulation operation unit that simulates the operation control, function transfer, and actual operation conditions of the educational equipment for education and provides it to the connected trainee terminal; Equipment management unit for managing equipment information required for education of educational equipment and operation information collected from the operation server of educational equipment; an augmented reality core unit that identifies an object from image information input from a trainee terminal and provides product information and education information that matches the object based on augmented reality; Education information DB that manages maintenance procedure scenarios, component information, training progress results, and user information for educational equipment; and a network unit for managing connections with the trainee terminal, the manager terminal, the object generator, and the operation server.

In the present invention, the augmented reality core unit recognizes the target object based on DCRI (Detection, Classification, Recognition, Identification) based on deep learning from image information input from the trainee terminal.

In the present invention, the augmented reality core unit recognizes an object using matrix decomposition, optimization by camera posture, and visual inertia measurement method based on image information and motion information input from the trainee terminal.

In the present invention, the augmented reality core unit extracts feature points by fusing edge, color and texture tracking from image information, and connects the corresponding points between frames based on the feature points to track the movement path. characterized.

Augmented reality-based education system according to an aspect of the present invention can perform operation and maintenance training for equipment while shooting control equipment based on augmented reality through a head mounted display or a tablet PC, so operation fails due to lack of experience can be reduced and maintenance efficiency can be increased.

1 is a block diagram illustrating an augmented reality-based education system according to an embodiment of the present invention.
2 is an exemplary screen of a trainee terminal in an augmented reality-based education system according to an embodiment of the present invention.
3 is a block diagram specifically illustrating an object generator of an augmented reality-based education system according to an embodiment of the present invention.
4 is a block diagram specifically illustrating an education server of an augmented reality-based education system according to an embodiment of the present invention.

Hereinafter, an augmented reality-based education system according to the present invention will be described with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

1 is a block diagram illustrating an augmented reality-based education system according to an embodiment of the present invention, and FIG. 2 is an exemplary screen of a trainee terminal in an augmented reality-based education system according to an embodiment of the present invention, FIG. 3 is a block diagram specifically showing an object generator of an augmented reality-based education system according to an embodiment of the present invention, and FIG. 4 is a detailed diagram of an education server of an augmented reality-based education system according to an embodiment of the present invention. It is a block diagram shown by .

1 to 4, the augmented reality-based education system according to an embodiment of the present invention includes a trainee terminal 10, a manager terminal 30, an object generator 40, and an education server 20 as shown in FIGS. may include

The trainee terminal 10 connects to the education server 20 based on UI (User Interface) / UX (User eXperience) so as to photograph the educational equipment and output educational information to perform operation and maintenance training of the educational equipment. As shown in FIG. 2 in conjunction with the education server 20, including the operating program, it is possible to demonstrate the educational procedure through arbitrary manipulation and monitor the educational equipment in real time.

In addition, the trainee terminal 100 can display the education information for training the operation and maintenance of the educational equipment so as not to overlap three-dimensionally with the object as the center, and according to the moving direction of the screen, the educational information displayed at the corresponding position is displayed. can be checked

The manager terminal 30 includes an operation control program to set an object for education and manage the educational situation, manages access and information of trainees, selects, proceeds and evaluates maintenance training, and manages the network state and monitor the progress of the training.

The object generator 40 may set an object for operation and maintenance training for educational equipment.

Here, the object generator 40 may include an object management unit 410 , an effect management unit 420 , a step management unit 430 , an education management unit 440 , and a network connection unit 450 as shown in FIG. 3 .

The object management unit 410 is for managing component information used for operation and maintenance training for educational equipment, and may include product information, component operation information, related information between components, and reaction information during component operation.

The effect management unit 420 is for expressing a reaction method when a component is manipulated, and may include component rotation and movement, component animation display, component color change, diode-type component blinking effect, and sis play display.

The step management unit 430 is for managing educational procedure scenarios, and may include normal/abnormal procedure scenario management, solution management according to procedures, connection of results according to procedure execution, and cyber security procedure scenario management.

The education management unit 440 is for managing the education progress result information, and may include a maintenance procedure execution result display, a malfunctioning maintenance procedure result record, and a score display according to the maintenance procedure performance result.

The network connection unit 450 is for managing a connection with the education server 20 and transmitting data, and accesses the education server 20 to update information, add or delete data, and exchange with the education server 20 . to convert the communication data.

The education server 20 is built based on a REST (Representational State Transfer)ful framework, extracts an object based on image information and motion information input from the trainee terminal 10, and provides educational information matching the object to the education information DB It is extracted from 240 and output to the trainee terminal based on augmented reality, and education information according to the education object is provided to the trainee terminal 10 selected from the manager terminal 30 to conduct education and manage the education situation, and an object generator Objects input from (40) can be set and stored in the education information DB (240) for management.

Here, the education server 20 may include a simulation operation unit 210 , an equipment management unit 220 , an augmented reality core unit 230 , an education information DB 240 , and a network unit 250 as shown in FIG. 4 . there is.

The simulation operation unit 210 simulates the operation control, function transfer, and actual operation situation of the educational equipment for education and provides it to the connected trainee terminal 10, and synchronizes the training procedures in the multi-connected trainee terminal 10. can be controlled

The equipment management unit 220 can manage the equipment information required for training of educational equipment and real-time operation information collected from the operation server (not shown) of the educational equipment, so that the administrator terminal 30 and the trainee terminal 10 request Information may be searched for and provided, and procedures for countering cyber security may be included.

The augmented reality core unit 230 may identify an object from image information input from the trainee terminal 10 and provide product information and education information that match the object based on augmented reality.

Here, the augmented reality core unit 230 may recognize a target object based on DCRI (Detection, Classification, Recognition, Identification) based on deep learning from image information input from the trainee terminal 10 .

At this time, the augmented reality core unit 230 performs optimization work by separately calculating matrix decomposition and camera posture to recognize an object based on image information and motion information input from the trainee terminal 10, and can cope with sudden movements. To compensate for missing cameras, Visual Inertial Odometry (VIO) can be used.

In addition, the augmented reality core unit 230 may track the movement path using a marker from the image information, and extract the feature point by fusion of edge, color and texture tracking without a marker, The movement path can be traced by using a specific pattern of the target by connecting the corresponding points between frames based on the feature point.

The education information DB 240 may manage a maintenance procedure scenario for educational equipment, component information, training progress results, and user information.

The network unit 250 may manage the connection with the trainee terminal 10 , the manager terminal 30 , the object generator 40 , and the operation server of the educational equipment.

In addition, the education server 20 may provide the information output to the trainee terminal 10 to the manager terminal 20 in a cybersecurity environment to monitor the same.

As described above, according to the education system according to an embodiment of the present invention, it is possible to perform operation and maintenance training for equipment while shooting control equipment based on augmented reality through a head mounted display or a tablet PC. Not only can the start failure caused by this be reduced, but also the efficiency of maintenance can be increased.

Implementations described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementations of the discussed features may also be implemented in other forms (eg, as an apparatus or program). The apparatus may be implemented in suitable hardware, software and firmware, and the like. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a computer, a microprocessor, a processing device, including an integrated circuit or programmable logic device, and the like. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDA”) and other devices that facilitate communication of information between end-users.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely an example, and those skilled in the art to which various modifications and equivalent other embodiments are possible. will understand

Accordingly, the true technical protection scope of the present invention should be defined by the following claims.

10: trainee terminal 20: education server
30: administrator terminal 40: object generator
210: simulation operation unit 220: equipment management unit
230: augmented reality core unit 240: education information DB
250: network unit 410: object management unit
420: effect management unit 430: step management unit
440: education management unit 450: network connection unit

Claims (6)

a trainee terminal for photographing educational equipment to display education information for operation and maintenance education of the educational equipment in three dimensions based on an object, and displaying the education information at a corresponding location according to a moving direction of the screen;
an administrator terminal for setting an object for education and managing the educational situation;
an object generator for setting an object for operation and maintenance training for the educational equipment; and
Extracting an object based on image information and motion information input from the trainee terminal, extracting education information matching the object from the education information DB, and outputting it to the trainee terminal based on augmented reality, and the trainee selected from the manager terminal An education server that provides the education information according to the education object to the terminal to conduct education and manage the educational situation, sets the object input from the object generator, and stores it in the education information DB for management; Augmented reality based education system.
The augmented reality-based system of claim 1, wherein the manager terminal comprises an operation control program for managing trainees, conducting and evaluating maintenance training, managing network status, and monitoring training progress. education system.
According to claim 1, wherein the object generator,
an object management unit for managing component information used in operation and maintenance training for the educational equipment;
an effect management unit for expressing a reaction method when the components are manipulated;
a staff management unit that manages the educational procedure scenario;
Education management department that manages education progress result information; and
Augmented reality-based education system comprising a; a network connection unit for managing a connection with the education server and transmitting data.
The augmented reality-based education system according to claim 1, wherein the education server provides information output to the trainee terminal to the manager terminal in a cybersecurity environment.
According to claim 1, wherein the education server,
a simulation operation unit for simulating operation control, function transfer, and actual operation of educational equipment for education and providing it to the connected trainee terminal;
an equipment management unit for managing equipment information necessary for education of the educational equipment and operation information collected from an operation server of the educational equipment;
an augmented reality core unit that identifies an object from the image information input from the trainee terminal and provides product information and the education information that match the object based on augmented reality;
an education information DB for managing maintenance procedure scenarios, component information, training progress results, and user information for the educational equipment; and
Augmented reality-based education system comprising a; a network unit for managing a connection with the trainee terminal, the manager terminal, the object generator, and the operation server.
The method of claim 5, wherein the augmented reality core unit extracts feature points by fusing edge, color, and texture tracking from the image information, and connects corresponding points between frames based on the feature points. Augmented reality-based education system, characterized in that tracking the movement path.

Images