KR20220041981A - Head mounted display based augmented reality - Google Patents

Abstract

According to the present invention, disclosed is a head-mounted display device based on augmented reality, which comprises: a camera module for photographing a front image of a user; a communication module for transmitting object information obtained from the camera module to a central server, and receiving content for am operation history and maintenance training transmitted from the central server; a display module for three-dimensionally displaying the content provided by the central server, and providing information on a specific location included in the content in accordance with voice of the user and screen movement; and an information processing module for identifying at least one object from the front image as a part of an operation of identifying an object corresponding to a gaze of the user, identifying the gaze of the user from a user image, determining the object gazed by the user among the objects as an object corresponding to the gaze of the user, on the basis of the gaze of the user, and displaying content corresponding to the object through the display module. Therefore, a user can receive, process, and display an image of an operation history and maintenance training for a desired specific facility.

Description

Augmented reality-based head mounted display device {HEAD MOUNTED DISPLAY BASED AUGMENTED REALITY}

The present invention relates to a display device for mounting a head based on augmented reality, and more particularly, it detects the movement and gaze of the user, and uses this to capture the equipment in the field based on augmented reality by using the information on the specific equipment desired by the user. It relates to an augmented reality-based head-mounted display device that can receive, process and display images such as operation history and maintenance training.

Recently, with the development of image players and image quality, interest in head mounted displays (HMDs) is gradually increasing.

A head wearable display (HMD) refers to various digital devices that are worn on the head like glasses to receive multimedia contents. According to the trend of weight reduction and miniaturization of digital devices, various wearable computers have been developed, and head-worn displays are also widely used. The head-worn display can provide various conveniences to users by combining with augmented reality technology and N-screen technology beyond a simple display function.

In addition, a user wearing the head mounted display may interact with the HMD in various situations. In this case, in order for the head-worn display to provide a better context-aware service to the user, a simpler user interface is required.

A recently developed head-wearable display device may provide virtual reality or augmented reality as an image to a user.

Most of these augmented reality services use a smartphone, and when an image of an area that a user wants to be augmented is acquired through a camera, specific information is superimposed inside/outside it using the feature points of the object, or, more realistically, a device After confirming the location using the GPS, gyro, and altitude sensor installed in the device, the location information is superimposed on the surrounding information using the distance information measured indirectly through the camera using the location as the starting point.

Recently, a simulation system for training of the distribution system has been built and used to increase the skill level of the operator for handling failure events.

That is, the training center is operated for the purpose of training to cope with the failure occurring on the basis of the measurement information (One Snap-Shot) measured on an arbitrary line.

The current distribution system operating system performs a function of remotely controlling and monitoring automation equipment to monitor failures occurring in the distribution system and perform line work.

The training simulation system for training the use of the current operating system generates an event (failure) and is configured in the form of training only the operator's ability to operate an automated facility to handle it.

It is necessary to improve proficiency by frequently training through operation and maintenance training for equipment while checking the control system directly in the field based on augmented reality by convergence with such a virtual simulation system.

The background technology of the present invention is disclosed in Korean Patent Application Laid-Open No. 10-2016-0009213 (published on January 26, 2016, a head mounted display and a content providing method using the same).

The present invention was created by the necessity as described above, and provides images such as operation history and maintenance education for a specific facility desired by the user while capturing the user's movement and gaze, and shooting the facility on the basis of augmented reality. An object of the present invention is to provide an augmented reality-based head mounted display device that can receive, process and display.

In order to achieve the above object, an augmented reality-based display device for mounting a head according to an aspect of the present invention, a camera module for photographing a user's front image, and the object information obtained from the camera module are transmitted to a central server, A communication module for receiving the contents for operation history and maintenance education transmitted from the central server, and displaying the contents provided by the central server in three dimensions, according to the user's voice and movement of the screen, the specific contents included in the contents As part of the operation of identifying a display module providing location information and an object corresponding to the user's gaze, identifying at least one object from the front image, identifying the user's gaze from the user image, and and an information processing module configured to determine an object that the user gazes at among at least one object based on the user's gaze as an object corresponding to the user's gaze and display the corresponding content through the display module. do.

In addition, in the present invention, the information processing module is characterized in that the camera module, the communication module, the display module and the memory and operatively connected (operatively coupled to).

In addition, the object provided by the content in the present invention is characterized in that it is at least one of field mode, education mode, DCS theory education, failure simulation, field control facility, EWS and OIS utilization, and educational video.

In addition, in the present invention, the information processing module displays the object in three dimensions through the display module, displays a UI for selecting at least one of the objects according to the user's head movement, and selects the selected object It is characterized in that the information matching with the request is provided to the central server and activated in the display module based on augmented reality.

In addition, in the present invention, the information processing module includes the field mode, the education mode, the DCS theory education, the failure simulation, the field control facility, the EWS and OIS utilization, and the contents corresponding to the object of the education image. It is characterized in that it is displayed in the form of a pop-up on the display module.

As described above, the augmented reality-based head-mounted display device according to an aspect of the present invention, unlike the prior art, recognizes the user's movement and gaze, and allows the user to capture the equipment in the field based on the augmented reality. Since it is possible to receive, process and display images such as operation history and maintenance training for a specific facility desired, users with insufficient experience can efficiently conduct maintenance and maintenance as well as training.

1 is a perspective view illustrating an augmented reality-based head mounted display device according to an embodiment of the present invention.
2 is a block diagram illustrating an augmented reality-based head mounted display device according to an embodiment of the present invention.
3 is a block diagram illustrating a process of generating and providing content in an augmented reality-based head mounted display device according to an embodiment of the present invention.
4 to 26 are exemplary screens for explaining contents generated and provided by an augmented reality-based head mounted display device according to an embodiment of the present invention.
27 is a block diagram illustrating an augmented reality education system to which an augmented reality-based head mounted display device is applied according to various embodiments of the present disclosure.
28 is a block diagram specifically illustrating an object generator of an augmented reality education system according to various embodiments of the present disclosure.
29 is a block diagram specifically illustrating an education server of an augmented reality education system according to various embodiments of the present disclosure.

Hereinafter, an augmented reality-based head mounted display device according to an embodiment of the present invention will be described with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation.

In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

1 is a perspective view illustrating an augmented reality-based head mounted display device according to an embodiment of the present invention, and FIG. 2 is an augmented reality based head mounted display device according to an embodiment of the present invention. 3 is a block diagram showing a process of generating and providing content in an augmented reality-based head mounted display device according to an embodiment of the present invention, and FIGS. 4 to 26 are an embodiment of the present invention This is an example screen for explaining the content generated and provided by the augmented reality-based head-mounted display device according to .

According to this embodiment, the augmented reality-based display apparatus 100 for mounting a head includes a camera module 110 , a communication module 120 , a display module 130 , an information processing module 140 , and an input module 160 . includes In this case, the augmented reality-based head mounting display device 100 may be provided in the form of glasses, and a fixing band 101 that can be stably fixed to the user's head on the temples of the glasses may be further provided.

The augmented reality-based head mounted display device 100 may acquire an image through the camera module 110 . For example, a front image obtained by photographing the front of the augmented reality-based head mounted display device 100 using the camera module 110 is acquired, and the object information acquired by the camera module 110 is converted into an information processing module It is processed in 140 and transmitted to the central server. At this time, the object information processed by the information processing module 140 is transmitted to the central server through the communication module 120 .

In addition, the augmented reality-based head mounted display device 100 may identify a specific object that the user gazes at from the front image acquired through the camera module 110 . For example, the augmented reality-based head mounted display device 100 may identify a specific object corresponding to the user's gaze by using the camera module 110 .

In addition, the augmented reality-based head mounted display apparatus 100 may map content to a specific object corresponding to the user's gaze. For example, the augmented reality-based display device for head mounting 100 receives an input (eg, voice) from a user through the input module 160 (input button, microphone, etc.), and generates content corresponding to the input. And, the generated content may be mapped to a specific object corresponding to the user's gaze.

According to various embodiments, when there is content mapped to a specific object corresponding to the user's gaze, the augmented reality-based display apparatus 100 for head mounting provides the mapped content to the user through the display 140 . can For example, the augmented reality-based head-mounted display device 100 may output the content through the display 140 so that the content mapped to a specific object is reflected on the glass (not shown) worn by the user. there is.

The augmented reality-based display device 100 for mounting a head according to this embodiment transmits the object information acquired from the camera module to the central server through the communication module 120, and transmits the operation history and maintenance education transmitted from the central server. receive content for Such contents are displayed in three dimensions through the display module 130 .

For example, the display module 130 displays the content provided by the central server in three dimensions under the control of the information processing module 140, and displays information on a specific location included in the content according to the user's voice and screen movement. to provide. Here, the objects provided by the content may be field mode, training mode, DCS theory training, failure simulation, field control equipment, EWS and OIS utilization, and training images.

The information processing module 140 according to the present embodiment may control a plurality of hardware or software components connected to the information processing module 140 by driving an operating system or an application, and may perform various data processing and operations. . For example, the information processing module 140 may be implemented as a system on chip (SoC). The information processing module 140 may load and process instructions or data received from at least one of the other components into the memory 150 , and store various data in the memory 150 .

In addition, the information processing module 140 displays the object in three dimensions through the display module 130, displays a UI for selecting at least one of the objects according to the user's head movement, and displays the selected object and Matching information is requested and provided to the central server and displayed on the display module 130 based on augmented reality.

As an example, referring to FIGS. 4 to 26 , the information processing module 140 may map content to an object that the user gazes at. For example, when a specified input (eg, a voice command or button input requesting content mapping, etc.) is received after the object the user is gazing at is identified, the information processing module 140 converts the received voice into text and then , can be mapped to the transformed text and the object the user is looking at.

Here, the information processing module 140 may identify at least one object from the front image by analyzing the front image. For example, the information processing module 140 may extract a feature point from the front image, and identify at least one object included in the front image based on the extracted feature point.

Next, an augmented reality education system using an augmented reality-based head mounted display device according to the present invention will be described with reference to FIGS. 27 to 29 .

27 is a block diagram illustrating an augmented reality education system to which an augmented reality-based head mounted display device is applied according to various embodiments of the present disclosure, and FIG. 28 is an augmented reality education system according to various embodiments of the present disclosure. It is a block diagram specifically showing an object generator of , and FIG. 29 is a block diagram specifically showing an education server of an augmented reality education system according to various embodiments of the present invention.

The augmented reality-based education system according to an embodiment of the present invention may include a trainee terminal 100 , a manager terminal 30 , an object generator 40 , and an education server 20 . Here, the trainee terminal 100 may be an augmented reality-based head mounted display device.

The trainee terminal 100 connects to the education server 20 based on UI (User Interface) / UX (User eXperience) so as to photograph the educational equipment and output educational information to perform operation and maintenance training of the educational equipment. Including the operating program, it is possible to realize the training procedure through arbitrary manipulation in conjunction with the training server 20 and monitor the training equipment in real time.

In addition, the trainee terminal 100 can display the education information for the operation and maintenance education of the educational equipment so as not to overlap in three dimensions, up, down, left, and right around the object, and the educational information displayed at the corresponding position according to the moving direction of the screen can be checked.

The manager terminal 30 includes an operation control program to set an object for education and manage the educational situation, manages access and information of trainees, selects, proceeds and evaluates maintenance training, and manages the network state and monitor the progress of the training.

The object generator 40 may set an object for operation and maintenance training for educational equipment.

Here, the object generator 40 may include an object management unit 410 , an effect management unit 420 , a step management unit 430 , an education management unit 440 , and a network connection unit 450 .

The object management unit 410 is for managing component information used for operation and maintenance training for educational equipment, and may include product information, component operation information, related information between components, and reaction information during component operation.

The effect management unit 420 is for expressing a reaction method when a component is manipulated, and may include component rotation and movement, component animation display, component color change, diode-type component blinking effect, and sis play display.

The step management unit 430 is for managing educational procedure scenarios, and may include normal/abnormal procedure scenario management, solution management according to procedures, connection of results according to procedure execution, and cyber security procedure scenario management.

The education management unit 440 is for managing the education progress result information, and may include a maintenance procedure execution result display, a malfunctioning maintenance procedure result record, and a score display according to the maintenance procedure performance result.

The network connection unit 450 is for managing a connection with the education server 20 and transmitting data, and accesses the education server 20 to update information, add or delete data, and exchange with the education server 20 . to convert the communication data.

The education server 20 is built based on a REST (Representational State Transfer)ful framework, extracts an object based on image information and motion information input from the trainee terminal 100, and provides educational information matching the object to the education information DB Extract from 240 and output to the trainee terminal based on augmented reality, and provide education information according to the education object to the trainee terminal 100 selected from the manager terminal 30 to conduct education and manage the education situation, object generator Objects input from (40) can be set and stored in the education information DB (240) for management.

Here, the education server 20 may include a simulation operation unit 210 , an equipment management unit 220 , an augmented reality core unit 230 , an education information DB 240 , and a network unit 250 .

The simulation operation unit 210 provides to the connected trainee terminal 100 by simulating the operation control, function transfer, and actual operation situation of the educational equipment for education, and synchronizes the training procedures in the multi-connected trainee terminal 100 . can be controlled

The equipment management unit 220 can manage the equipment information required for training of educational equipment and real-time operation information collected from the operation server (not shown) of the educational equipment, so that the administrator terminal 30 and the trainee terminal 100 request Information may be searched for and provided, and procedures for countering cyber security may be included.

The augmented reality core unit 230 may identify an object from the image information input from the trainee terminal 100 and provide product information and education information that match the object based on augmented reality.

Here, the augmented reality core unit 230 may recognize a target object based on DCRI (Detection, Classification, Recognition, Identification) based on deep learning from image information input from the trainee terminal 100 .

At this time, the augmented reality core unit 230 performs optimization work by separately calculating matrix decomposition and camera posture to recognize an object based on image information and motion information input from the trainee terminal 100, and can cope with sudden movements. To compensate for missing cameras, Visual Inertial Odometry (VIO) can be used.

In addition, the augmented reality core unit 230 may track the movement path using a marker from the image information, and extract the feature point by fusion of edge, color and texture tracking without a marker, The movement path can be traced by using a specific pattern of the target by connecting the corresponding points between frames based on the feature point.

The education information DB 240 may manage a maintenance procedure scenario for educational equipment, component information, training progress results, and user information.

The network unit 250 may manage the connection with the trainee terminal 100 , the manager terminal 30 , the object generator 40 , and the operation server of the educational equipment.

In addition, the education server 20 may provide the information output to the trainee terminal 100 to the manager terminal 20 in a cybersecurity environment to monitor the same.

As described above, according to the education system according to an embodiment of the present invention, it is possible to perform operation and maintenance training for equipment while shooting control equipment based on augmented reality through a head mounted display or a tablet PC. Not only can the start failure caused by this be reduced, but also the efficiency of maintenance can be increased.

Implementations described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementations of the discussed features may also be implemented in other forms (eg, as an apparatus or program). The apparatus may be implemented in suitable hardware, software and firmware, and the like. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a computer, a microprocessor, a processing device, including an integrated circuit or programmable logic device, and the like. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDA”) and other devices that facilitate communication of information between end-users.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely an example, and those skilled in the art to which various modifications and equivalent other embodiments are possible. will understand

Therefore, the true technical protection scope of the present invention should be defined by the following claims.

10: trainee terminal 20: education server
30: administrator terminal 40: object generator
100: Augmented reality-based head mounted display device
110: camera module 120: communication module
130: display module 140: information processing module
150: memory 160: input module
210: simulation operation unit 220: equipment management unit
230: augmented reality core unit 240: education information DB
250: network unit 410: object management unit
420: effect management unit 430: step management unit
440: education management unit 450: network connection unit

Claims (3)

a camera module for photographing a user's front image;
a communication module for transmitting the object information obtained from the camera module to a central server and receiving contents for operation history and maintenance education transmitted from the central server;
a display module that displays the content provided by the central server in three dimensions and provides information on a specific location included in the content according to a user's voice and screen movement; and
As part of the operation of identifying an object corresponding to the user's gaze, at least one object is identified from the front image, the user's gaze is identified from the user image, and at least one object is identified based on the user's gaze. an information processing module that determines the object the user gazes at as an object corresponding to the user's gaze and displays the corresponding content through the display module;
The information processing module is an augmented reality-based head mounted display device, characterized in that it is operatively coupled to the camera module and the communication module, the display module and the memory.
The method of claim 1,
The object provided in the content is any one or more of field mode, education mode, DCS theory education, failure simulation, field control facility, EWS and OIS utilization, and educational video,
The information processing module displays the object in three dimensions through the display module, displays a UI for selecting at least one of the objects according to the user's head movement, and displays information matching the selected object Augmented reality-based head-mounted display device, characterized in that it is provided by requesting the central server and activated in the display module based on the augmented reality.
3. The method of claim 2,
The information processing module pops up contents corresponding to the object of the field mode, the education mode, the DCS theory education, the failure simulation, the field control facility, the EWS and OIS utilization, and the educational image to the display module. Augmented reality-based head-mounted display device, characterized in that it is displayed in the form.

Images