KR102268797B1 - Syetem and method for providing augmented reality user interface - Google Patents

Abstract

The present invention relates to a system and method for providing augmented reality. The augmented reality providing method may include photographing an image including an object, recognizing an object included in the photographed image, receiving information on at least one component included in the object, the at least Receiving state information for one component, determining whether an event has occurred for each of the at least one component based on the received state information, augmented reality for the component in which the event occurred Receiving information, and generating and displaying an augmented reality interface based on the augmented reality information.

Description

Augmented reality providing system and method {Syetem and method for providing augmented reality user interface}

The present invention relates to a system and method for providing augmented reality. Specifically, it relates to an augmented reality providing system and method for providing augmented reality information on components included in a device being photographed.

Augmented reality (AR) refers to a computer graphics technique in which a virtual object or information is synthesized in a real environment to appear as an object existing in the original environment.

Unlike conventional virtual reality, which targets only virtual space and objects, augmented reality has a feature of providing additional information that is difficult to obtain only in the real world by synthesizing virtual objects on the basis of the real world. Because of these characteristics, augmented reality can be applied to a variety of real environments, unlike existing virtual reality, which can be applied only to fields such as games. In particular, it is spotlighted as a next-generation display technology suitable for a ubiquitous environment.

Since research on augmented reality technology has been started, research to utilize augmented reality technology has continued in various fields. Recently, with the spread of smartphones, augmented reality technology has been widely used for location-based services.

In general, augmented reality technology displays information about a device currently being photographed on a preview screen in the form of a menu. In this case, the preview screen may display status information of the device being photographed in 2D or 3D form.

However, in the case of the conventional augmented reality providing method, only the same menu for the target device is provided regardless of the type and state of the components included in the device being photographed. Therefore, when a plurality of devices of the same type are arranged, the same menu for each device is provided in augmented reality, and it is difficult to grasp information on the types or operating states of components included in the currently photographed device. there was

An object of the present invention is to provide an augmented reality providing system and method for displaying different augmented reality interfaces according to types and operating states of components included in a device being photographed.

In addition, the present invention provides an augmented reality providing system and method for determining whether each abnormal operation is present for a plurality of components included in the device being photographed, and displaying different augmented reality interfaces according to the cause of the abnormal operation aim to do

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. Moreover, it will be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

The augmented reality providing method of the present invention for achieving the above object includes the steps of (a) photographing an image including an object, (b) recognizing an object included in the photographed image, and (c) receiving information on at least one component, (d) receiving status information on the at least one component, based on the received status information, to each of the at least one component Step (e) of determining whether an event has occurred, (f) generating an augmented reality interface based on augmented reality information about the component in which the event has occurred, and adding the augmented reality interface to the captured image (g) of overlapping and displaying on the screen.

In addition, the step (e) of determining whether an event has occurred for each of the at least one component based on the received status information includes the status information of a specific component among the at least one component and a predetermined normal condition. (e1) determining whether the specific component operates normally by comparing the operation reference value, and if the specific component is not operating normally, determining the cause of the abnormal operation based on the status information, and It may include a step (e2) of generating an event including the cause of the operation.

Also, steps (e1) and (e2) may be repeatedly performed for each of the at least one component included in the object.

In addition, the step (f) of generating an augmented reality interface based on the augmented reality information on the component in which the event has occurred may include generating different augmented reality interfaces according to the cause of the abnormal operation included in the event. It may include receiving the augmented reality information for.

In addition, the state information may include power amount information, temperature information, voltage information, current information, or a cut-off state for the component.

In addition, the steps (b) or (f) may be performed in the server, and other remaining steps may be performed in the user terminal.

In addition, the step (f) of generating an augmented reality interface based on the augmented reality information on the component in which the event occurred includes generating different augmented reality interfaces according to the cause of the abnormal operation included in the event. And, the step (g) of overlapping the augmented reality interface on the captured image and displaying it on the screen may include displaying the generated augmented reality interface by overlapping the component in which the event occurred.

Also, the different augmented reality interfaces may include augmented reality modeling of different colors or different shapes.

In addition, the step (b) of recognizing the object included in the photographed image may include selecting the object included in the image by using location information at which the image was captured, a marker of the object, or markerless information of the object. This may include recognizing

Augmented reality providing system of the present invention for achieving the above object, a camera unit for capturing an image including an object, an object recognition unit for recognizing the object included in the image, at least one component included in the object A communication unit configured to receive information on an element, state information on the at least one component, and augmented reality information on the component in which an event has occurred, augmented reality for generating an augmented reality interface based on the augmented reality information A processing unit, a display unit displaying the augmented reality interface overlapping the captured image, and a control unit determining whether the event occurs for each of the at least one component based on the received state information.

In addition, the controller compares the state information of a specific component among the at least one component with a predetermined normal operation reference value to determine whether the specific component operates normally, and determines whether the specific component is operating normally. If not, determine the cause of the abnormal operation based on the state information, generate an event including the cause of the abnormal operation, select the component in which the event occurred, and the augmented reality for the selected component You can request information.

Also, the controller may request the augmented reality information for generating different augmented reality interfaces according to the abnormal operation included in the event.

In addition, the augmented reality processing unit generates different augmented reality interfaces depending on the cause of the abnormal operation included in the event, and the display unit displays the generated augmented reality interface by overlapping the component in which the event occurred. can do.

In addition, it may include a user terminal including the camera unit, the communication unit, the display unit and the control unit, and a server including the object recognition unit and the augmented reality processing unit.

The augmented reality providing system and method of the present invention allow the user to grasp the state of the device at a glance by expressing the types and operating states of components included in the device being photographed in augmented reality. Through this, the user can intuitively and easily grasp status information for each component included in the device. In addition, the convenience of operating the device may be improved.

In addition, the augmented reality providing system and method of the present invention determine whether abnormal operation of a plurality of components included in the device being photographed, respectively, and provide different augmented reality interfaces according to the cause of the abnormal operation. Through this, the present invention can shorten the time required for device management to increase user convenience and improve work processing speed. In addition, since it is possible to quickly process the work within the same time, work efficiency can be improved.

In addition to the above-described effects, the specific effects of the present invention will be described together while describing specific details for carrying out the invention below.

1 is a diagram illustrating an augmented reality providing system according to an embodiment of the present invention.
2 is a block diagram illustrating an apparatus for providing augmented reality according to an embodiment of the present invention.
3 is a diagram for explaining a method of operating an apparatus for providing augmented reality according to an embodiment of the present invention.
4 is a flowchart illustrating a method for providing augmented reality according to an embodiment of the present invention.
5 is a flowchart specifically explaining an example of a step of determining whether an event has occurred for each of the components of FIG. 4 .
6 is a flowchart for specifically explaining an example of a step of generating and displaying the augmented reality interface of FIG. 4 .
7 and 8 are diagrams for explaining an example of the operation of the augmented reality providing method according to an embodiment of the present invention.

The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

Hereinafter, an augmented reality providing system and method according to the present invention will be described with reference to FIGS. 1 to 8 .

1 is a diagram illustrating an augmented reality providing system according to an embodiment of the present invention.

Referring to FIG. 1 , the augmented reality providing system according to an embodiment of the present invention includes a user terminal 100 , a target device 200 , and a server 300 .

The user terminal 100 may operate as an augmented reality providing device. The user terminal 100 photographs the target device 200 and receives information on at least one component included in the target device 200 from the server 300 .

Next, the user terminal 100 receives status information on at least one component included in the target device 200 , and determines whether each of the at least one component is abnormal based on the status information.

Then, the user terminal 100 receives the augmented reality information on the component in which the event has occurred from the server 300, generates an augmented reality interface based on the augmented reality information, and displays it on the screen. Here, the augmented reality information includes augmented reality related data of components included in the captured target device 200 for generating the augmented reality interface.

Augmented reality information may be modeled and rendered as a 2D or 3D image, and the rendered image is converted into an augmented reality interface and displayed in a preview form on the screen of the user terminal 100 .

The user terminal 100 may include a mobile-based device, a wearable device (eg, a head mounted device (HMD), a video see-through device), It may be any one of an optical see-through device, a screen-based device, and a projector-based device.

In addition, the user terminal 100 is a computer, an Ultra Mobile PC (UMPC), a workstation, a net-book, a PDA (Personal Digital Assistants), a portable computer, a web tablet (web tablet), a wireless phone ( It may be a device capable of transmitting and receiving information such as a wireless phone, a mobile phone, a smart phone, an e-book, and a portable multimedia player (PMP) in a wireless environment. However, the present invention is not limited thereto.

The target device 200 may be various electronic devices including at least one component. For example, the target device 200 may be an automatic control device, an industrial device, or a switchboard. In this case, the target device 200 may include a circuit breaker, a relay, a communication link device, a watt-hour meter, a sensor unit, a human machine interface (HMI), etc. as components. A description of each component is sufficiently well known, a detailed description thereof will be omitted herein.

As another example, the target device 200 may be a measurement target of an automatic meter reading system (hereinafter, AMR). In general, the remote meter reading system (AMR) refers to converting a measurement result into an electrical signal from a measurement target at a distant point, transmitting data through a communication network, and processing the data in a computer.

The server of the remote meter reading system (AMR) uses a data communication network (eg, backbone network, private network, Internet network included) provided to perform data communication based on TCP/IP (Transmission Control Protocol/Internet Protocol). interconnected with the collection device. The server performs data transmission and reception in real time with the data collection device through the data communication network, and it is possible to monitor the power state of each measurement target in real time.

In addition, the server receives the meter reading data of each measurement target from the data collection device in real time. Based on the meter reading data, the power status of each measurement target, for example, power failure, conduction, and leakage status is analyzed and provided to the manager.

Here, the meter reading data may be the sum total of the amount of power used by all measurement targets allocated to one data concentrator device, the amount of power used by each measurement target, the instantaneous power amount, etc. In addition, all data that can determine the power quality of each measurement target may be included.

The server 300 includes a database for storing various information about the target device 200 . For example, the server 300 includes object information on the target device 200 , information on components included in the target device 200 , status information of each component, and the target device 200 and each component. Augmented reality information about the may be stored.

The server 300 may operate as a data collection device or server of the remote meter reading system (AMR). That is, the server 300 may receive and store the meter reading data for the target device 200 in real time. Through this, the server 300 may monitor and remotely control the operating state of the target device 200 . However, the present invention is not limited thereto.

2 is a block diagram illustrating an apparatus for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 2 , the user terminal 100 as an augmented reality providing apparatus according to an embodiment of the present invention includes a controller 110 , a camera unit 120 , an object recognition unit 130 , and a location measurement unit 140 . , an augmented reality processing unit 150 , a communication unit 160 , a sensor unit 165 , a memory unit 170 , a display unit 180 , and an interface unit 190 . However, since the components of the user terminal 100 shown in FIG. 2 are not essential, the user terminal 100 having more components or fewer components may be implemented.

The controller 110 may control the overall operation of the components included in the user terminal 100 . The controller 110 may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the controller 350 may access, store, manipulate, process, and generate data in response to the execution of the software. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media. A detailed operation of the control unit 110 will be described below.

The camera unit 120 is a device for capturing an image, and the captured image is transmitted to the controller 110 . The image captured by the camera unit 120 may be displayed on the display unit 180 in the form of a real-time preview.

The object recognition unit 130 recognizes an object included in the image captured by the camera unit 120 . The object recognition method used by the object recognition unit 130 includes a marker-based recognition method for recognizing an object using a marker included in an image, and a markerless recognition method for recognizing a pre-registered real object instead of a physical marker.

Specifically, the marker-based recognition method is a method of recognizing an object based on a marker such as a QR code, a data matrix, or a 2D barcode. The marker-based recognition method utilizes a special type of marker that the system understands in advance, so the calculation speed and recognition rate are high.

The markerless recognition method is divided into a natural image-based method and an environment recognition-based method. The natural image-based method analyzes a number of feature points existing in a general flat image and uses the analyzed feature points as invisible markers. The natural image-based method uses a technique for extracting a feature point existing in a natural image and a technique for effectively converting the feature point into a descriptor. The environment recognition-based method is a technique for recognizing the environment itself in the form of a three-dimensional map without special markers or pre-input images. In the environment recognition-based method, a mapping technology that creates a three-dimensional environment map in real time by analyzing data from a camera or RGBD sensor, and a camera tracking technology that estimates the current camera position on the created map can be used. have.

The object recognition unit 130 may recognize an object included in the captured image using the marker-based recognition method and the markerless recognition method described above. In this case, the object recognition unit 130 may recognize an object included in the image based on location information at which the image was captured, a marker for the object, or markerless information of the object. However, the method of recognizing the object by the object recognition unit 130 is not limited thereto, and various newly developed methods may be applied.

However, the operation of the object recognition unit 130 may be performed in a separate external device (eg, the server 300 ), and only result data for the execution result may be received and used by the user terminal 100 . .

The position measuring unit 140 measures the current position of the user terminal 100 and the position of a photographed object. Specifically, the location measurement unit 140 may measure the current location of the user terminal 100 and the location of an object photographed by the user terminal 100 using a local location tracking technology. Depending on the method used, local location tracking technology may include time of flight (TOF), spatial scan, inertial sensing, mechanical linkage, phase difference sensing, and direct field detection. (direct field sensing), and a mixture of the above techniques can be used to increase accuracy.

The location measurement unit 140 may measure a location using a location-based service using near field communication (NFC). For example, the location-based service may use Zigbee and iBeacon technologies. The location-based service can be mainly used in an indoor environment where GPS cannot be used, and the relative location can be determined through the signal strength of each node. In addition, the position measurement unit 140 may increase the accuracy of position measurement by using various sensors (inertial measurement sensor, altimeter, compass) installed in the user terminal 100 .

The augmented reality processing unit 150 may generate an augmented reality interface provided to the user based on the augmented reality information. Augmented reality information is raw data used to model and render a photographed object. The augmented reality information may be stored in a database of the server 300 or stored in the memory unit 170 of the user terminal 100 and used.

The augmented reality processing unit 150 may use a modeling technique for accurately expressing a real object in virtual reality based on the augmented reality information, and a rendering technique for realistically expressing the real object. The modeling technique is a technique for synthesizing a 2D image input as a video by reflecting depth information. Modeling technology enables various applications that cannot be achieved in real space by performing object transformation and color change according to the user's intention. Rendering technology is a technology that allows a user to feel a sense of vitality in an augmented reality interface by realistically expressing the virtual model by adding lighting effects, shadow effects, and brightness changes to the generated virtual models.

However, the operation of the augmented reality processing unit 150 may be performed in a separate external device (eg, the server 300 ), and only result data of the execution result may be received and used by the user terminal 100 . .

The communication unit 160 exchanges data with the server 300 that stores and manages the augmented reality database or the target device 200 that is a photographing target. In this case, the augmented reality database is an augmented reality object and a database storing attribute information of the augmented reality object.

The communication unit 160 may include an electronic device, module, or program that transmits and receives data to and from another terminal device in a wired or wireless manner. The communication protocol used by the communication unit 160 includes, for example, CDMA (Code Division Multiple Access) communication, WCDMA (Wideband Code Division Multiple Access) communication, or broadband wireless communication. However, the present invention is not limited thereto.

The sensor unit 165 is a device for detecting information necessary for augmented reality or context information additionally applied to augmented reality. The sensor unit 165 may include a temperature sensor, a humidity sensor, a position sensor, a motion sensor, and the like. Here, a GPS sensor that detects a GPS signal may be used as the location sensor, and a gyroscope sensor or an acceleration sensor may be used as the motion sensor. In addition, the sensor unit 165 may measure the current position, movement, tilt, direction, distance from the object to be photographed, ambient sound, brightness, and the like of the user terminal 100 .

The memory unit 170 stores an operating system for controlling the overall operation of the user terminal 100 , an application program, and data for storage (a phone number, an SMS message, a compressed image file, a video, etc.). In addition, the memory unit 170 may store the augmented reality object corresponding to the object to be photographed, attribute information and state information of the augmented reality object. That is, the memory unit 170 stores the object to be photographed, state information about the components included in the object, and augmented reality information.

The memory unit 170 may include a random access memory (RAM) or read only memory (ROM) and a static storage device for storing static information and instructions. The memory unit 170 may be a volatile memory unit or a non-volatile memory unit. Memory unit 170 may also be another type of computer readable medium, such as a magnetic or optical disk.

The display unit 180 displays status information (or indicators) generated during operation of the user terminal 100 , limited numbers and characters, moving pictures and still pictures, and the like. Also, the display unit 180 may display an image received through the camera unit 120 and display an augmented reality interface corresponding to a photographed object.

The interface unit 190 may receive an input from a user. The interface unit 190 may include an input device such as a button type input unit, a touch screen, a keyboard, or a mouse. For example, the touch screen may be formed on the upper surface of the display unit 180 and may receive an input through a user's touch. However, the present invention is not limited thereto.

3 is a diagram for explaining a method of operating an apparatus for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 3 , the user terminal 100 , which is an augmented reality providing apparatus according to an embodiment of the present invention, captures a target device 200 , and displays an augmented reality interface for the captured target device 200 on the display unit ( 190) is displayed.

Here, the target device 200 may include first to fourth components 210 , 220 , 230 , and 240 . For example, the target device 200 may be an automatic control device, an industrial device, or a switchboard. In this case, the first to fourth components 210 , 220 , 230 , and 240 may be any one of a circuit breaker, a relay, a communication link device, a watt-hour meter, a sensor unit, and an HMI, respectively. However, the present invention is not limited thereto.

The server 300 works with the target device 200 to store various data about the target device 200 . The server 300 includes an object information DB 310 , a component DB 320 , a state information DB 330 , and an augmented reality DB 340 . Hereinafter, the target device 200 will be described by mixing it with the term 'object', which is a photographing target.

The object information DB 310 includes various information about objects. For example, it may include information such as product information (product name, product category, installation time, management method, etc.) of the object, location information where the object is installed, and a circuit diagram and installation diagram related to the object.

The component DB 320 includes information on all components included in the object. The component DB 320 includes information on components included in each object, a connection relationship between each component, a predetermined normal operating range of each component, and replaceable components in case of failure.

For example, in the case of a circuit breaker included in a switchboard, it includes VCB (Vacuum Circuit Breaker), ACB (Air Circuit Breaker), MCCB (Molded　Case Circuit Breaker, circuit breaker for wiring), etc., and component DB 320 may include information about the type of circuit breaker included in the target distributor and replaceable circuit breaker.

The state information DB 330 includes operation information of each component included in the object. For example, the state information DB 330 includes wattage information, temperature information, voltage information, current information, or state information of each component (eg, cut-off state or trip state) for the components of each object. can do. Status information of each component may be updated in real time, and may be used as a part of a remote meter reading system (AMR).

The augmented reality DB 340 stores augmented reality information that is raw data used to model and render an object. In addition, the augmented reality DB 340 stores the augmented reality interface provided to the user based on the augmented reality information. That is, the augmented reality DB 340 stores both the augmented reality information before modeling and rendering of the object, and the augmented reality interface after modeling and rendering, and may provide necessary data according to the request of the user terminal 100 . .

The user terminal 100 recognizes the target device 200 as an object included in the captured image, and augments the first to fourth components 210 , 220 , 230 , and 240 included in the target device 200 . Reality information or augmented reality interface is received from the server 300 and displayed on the screen. In the augmented reality interface, augmented reality icons for the first to fourth components 210 , 220 , 230 , and 240 that are not actually visible may be expressed to overlap on the target device 200 . However, this is only an example, and the present invention is not limited thereto.

A detailed description of the augmented reality providing method performed in the user terminal 100 will be described below.

4 is a flowchart illustrating a method for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 4 , in the augmented reality providing method according to an embodiment of the present invention, an image including an object is first captured by the camera unit 120 of the user terminal 100 ( S110 ).

Next, the object recognition unit 130 recognizes an object included in the captured image (S120).

Next, the control unit 110 receives information on at least one component included in the object through the communication unit 160 (S130). The controller 110 may receive information on the recognized object and information on components included in the recognized object from the server 300 . In addition, when information on the corresponding object has been received in the past, the information on the object may be received from the memory unit 170 that stores the information of the corresponding object. In this case, the component included in the object may be a plurality.

Next, the control unit 110 receives the state information on at least one component included in the object from the server 300 through the communication unit 160 (S140). The server 300 stores state information of each component included in the object in real time based on the remote meter reading system (AMR). The controller 110 may request and receive current state information of each component stored in the server 300 .

Next, the control unit 110 determines whether an event has occurred for each of the at least one component based on the received state information (S150). Here, the event may occur when there is an abnormality in the operation of the corresponding component. The event may include information on the cause of the abnormal operation of the corresponding component.

Then, the control unit 110 receives the augmented reality information on the component in which the event is generated through the communication unit 160 (S160). The controller 110 may receive augmented reality information of a corresponding component from the server 300 .

Next, the augmented reality processing unit 150 generates an augmented reality interface based on the received augmented reality information, and displays the augmented reality interface by overlapping the image captured by the display unit 180 ( S170 , S180 ).

Additionally, in another embodiment of the present invention, the control unit 110 receives the augmented reality interface stored in advance in the server 300 for the corresponding component, instead of receiving the augmented reality information on the component in which the event has occurred, and the display unit (180) can be displayed directly.

For example, the server 300 may perform the operations of the object recognition unit 130 and the augmented reality processing unit 150 of the user terminal 100 described above. Subsequently, the user terminal 100 may receive and use result data of the operation performed from the server 300 . Specifically, the user terminal 100 transmits the image captured by the camera unit 120 to the server 300 , and the server 300 recognizes an object included in the received image, and augments reality for the recognized object. An interface may be created and provided to the user terminal 100 .

In this case, the user terminal 100 directly outputs the data (ie, the augmented reality interface) received from the server 300 to the display unit 180 . Accordingly, since the amount of calculation of the control unit 110 is reduced, the operation speed may be increased.

5 is a flowchart specifically explaining an example of a step of determining whether an event has occurred for each of the components of FIG. 4 .

Referring to FIG. 5 , determining whether an event has occurred for each component included in the object in step S150 of FIG. 4 may be performed by the controller 110 in the following order.

First, any one of at least one component included in the object is specified (S251).

Next, by comparing the state information of the specified component with a predetermined normal operation reference value, it is determined whether the specified component is operating normally ( S252 , S253 ). For example, the state information includes wattage information, temperature information, voltage information, or current information for a component. The normal operation reference value is a normal operation range for each state numerical value, and may be stored in advance in the memory unit 170 and used.

Next, when it is determined that the specified component is not operating normally, the control unit 110 determines the cause of the abnormal operation based on the state information of the specified component ( S255 ). For example, when the temperature of the specified component is out of the normal operating range, it may be determined that the temperature abnormal occurs, and when the amount of power of the specified component is out of the normal operating range, it may be determined that the electrical energy abnormal is generated.

Next, the control unit 110 generates an event including the determined cause of the abnormal operation (S256). Information on the component in which the event has occurred may be transmitted to the augmented reality processing unit 150 or stored in the memory unit 170 .

Then, the control unit 110 changes the specific component to another component (S254).

Next, the control unit 110 repeats steps S251 to S256 for other components. Through this, it can be determined whether all components included in the object operate normally.

6 is a flowchart for specifically explaining an example of a step of generating and displaying the augmented reality interface of FIG. 4 .

Referring to FIG. 6 , generating and displaying the augmented reality interface in steps S170 and S180 of FIG. 4 may be performed by the augmented reality processing unit 150 in the following order.

First, the augmented reality processing unit 150 extracts the cause of the abnormal operation included in the generated event (S371). For example, when there is an abnormality in the temperature of the corresponding component, the augmented reality processing unit 150 may extract information on the corresponding component and information on the cause of the abnormal operation such as the occurrence of the temperature abnormality from the event.

Next, the augmented reality processing unit 150 generates different augmented reality interfaces according to the cause of the abnormal operation included in the event ( S373 ). In this case, the augmented reality processing unit 150 may request augmented reality information for generating different augmented reality interfaces from the server 300 . The augmented reality processing unit 150 may generate an augmented reality interface based on the received augmented reality information.

In this case, the augmented reality interface may be generated using different modeling and rendering images depending on the cause of the abnormal operation. In addition, the augmented reality interface may be expressed in different colors according to the degree of the abnormal operation.

For example, the augmented reality interface (eg, flame rendered image) generated when the cause of abnormal operation is temperature and the augmented reality interface (eg, rendered image of lightning) generated when the cause of abnormal operation is the amount of electricity can be different.

In addition, when the temperature of the component in which the abnormality occurs is 100 degrees and 150 degrees, it may be expressed in different colors. However, this is only an example, and the present invention is not limited thereto.

Next, the controller 110 overlaps the augmented reality interface generated by the augmented reality processing unit 150 on the corresponding component in which the event has occurred and displays it ( S380 ).

Additionally, in another embodiment of the present invention, the control unit 110 receives the augmented reality interface stored in advance in the server 300 for the corresponding component, instead of receiving the augmented reality information on the cause of the abnormal operation to the display unit 180 . can be displayed directly on In this case, the controller 110 may request an augmented reality interface for the cause of the abnormal operation from the server 300 .

7 and 8 are diagrams for explaining an example of the operation of the augmented reality providing method according to an embodiment of the present invention.

Referring to FIG. 7 , in the augmented reality providing method according to an embodiment of the present invention, the user terminal 100 photographs a target device 200 , and the target device 200 includes first to fourth components 210 , 220, 230, 240) may be included. The user terminal 100 may display the first to fourth components 210 , 220 , 230 , and 240 to overlap on the target device 200 in augmented reality.

At this time, the control unit 110, based on the state information of the first to fourth components 210, 220, 230, 240, the first and fourth components 210 and 240 are in normal operation, the second and It may be determined that an event has occurred in the third components 210 and 240 .

The augmented reality processing unit 150 may determine that a temperature abnormality has occurred in the second component 220 and a power quantity abnormality has occurred in the third component 230 based on the generated event.

The augmented reality processing unit 150 may overlap the first augmented reality interface (eg, a rendered image in which a flame burns and a description thereof) on the second component 220 in which the temperature abnormality occurs.

In addition, the augmented reality processing unit 150 may overlap the second augmented reality interface (eg, a rendered image of lightning and a description thereof) on the third component 230 in which the amount of power is abnormal.

Referring to FIG. 8 , the user may select, through the interface unit 190 of the user terminal 100 , portions of the second and third components 210 and 240 in which the first and second augmented reality interfaces overlap. .

In this case, the augmented reality processing unit 150 may generate and display an additional menu including specific information on the first and second augmented reality interfaces. Here, the additional menu may include detailed information related to the cause of each abnormality.

For example, when the cause of the abnormality is temperature, the augmented reality processing unit 150 generates information on the normal temperature range of the component, the temperature of the current component, and the cooling-down method of the component as an additional menu to create the first augmentation. It can be reflected in the real interface.

As another example, when the cause of the abnormality is the amount of power, the augmented reality processing unit 150 generates information about the normal power range of the component, the current amount of power, a graph representing the change in the amount of power from the past to the present, as an additional menu, and creates the second augmented reality It can be reflected in the interface. However, this is only an example, and the present invention is not limited thereto.

However, the present invention is not limited to generating an additional augmented reality interface for the target device 200 only for the first to fourth components 210 , 220 , 230 , and 240 . Additionally, according to the present invention, if there is additional notification information for a component, it may be generated and displayed as an augmented reality interface.

For example, when the replacement time of a component included in the target device 200 approaches, an augmented reality interface including information on a related replacement model may be generated and displayed on the corresponding component. However, the present invention is not limited thereto.

As described above, in the augmented reality providing system and method of the present invention, the type and operating state of each component included in the object being photographed may be expressed in augmented reality and provided to the user. Specifically, the present invention determines whether an abnormal operation is performed for a plurality of components included in an object being photographed, and provides different augmented reality interfaces according to the cause of the abnormal operation.

Accordingly, the user can intuitively and easily grasp status information for each component included in the device. That is, the present invention can shorten the time required for device management to increase user convenience and improve work processing speed. In addition, since it is possible to quickly process the work within the same time, work efficiency can be improved.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can realize that the present invention can be implemented in other specific forms without changing the technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: user terminal
200: target device
300: server

Claims (14)

As an augmented reality providing method performed in a user terminal or server,
(a) photographing an image including an object;
(b) recognizing an object included in the photographed image;
(c) receiving information on at least one component included in the object;
(d) receiving status information on the at least one component;
(e) determining whether an event has occurred for each of the at least one component based on the received state information;
(f) generating an augmented reality interface based on augmented reality information on the component in which the event has occurred; and
(g) overlapping the augmented reality interface on the captured image and displaying on the screen,
(e) determining whether an event has occurred for each of the at least one component based on the received state information,
(e1) determining whether the specific component operates normally by comparing the state information of the specific component among the at least one component with a predetermined normal operation reference value;
(e2) determining the cause of the abnormal operation based on the state information when it is determined that the specific component is not operating normally, and generating an event including the cause of the abnormal operation
How to provide augmented reality.
delete According to claim 1,
The step (e1) and the step (e2) are repeatedly performed for each of the at least one component included in the object.
According to claim 1,
(f) of generating an augmented reality interface based on the augmented reality information on the component in which the event has occurred,
and receiving the augmented reality information for generating different augmented reality interfaces according to a cause of the abnormal operation included in the event.
According to claim 1,
The state information is, the augmented reality providing method including the amount of power information, temperature information, voltage information, current information, or a blocking state for the component.
According to claim 1,
The step (b) or (f) is performed in the server, and the other steps are performed in the user terminal.
According to claim 1,
The step (f) of generating an augmented reality interface based on the augmented reality information on the component in which the event has occurred includes generating different augmented reality interfaces according to the cause of the abnormal operation included in the event,
The step (g) of overlapping the augmented reality interface on the captured image and displaying the augmented reality interface on the screen includes overlapping the generated augmented reality interface with the component in which the event has occurred and displaying the overlapped.
8. The method of claim 7,
The different augmented reality interfaces include augmented reality modeling of different colors or different shapes.
According to claim 1,
Step (b) of recognizing an object included in the photographed image,
Augmented reality providing method comprising recognizing the object included in the image by using the location information at which the image was captured, the marker of the object, or the markerless information of the object.
a camera unit for capturing an image including an object;
an object recognition unit for recognizing the object included in the image;
a communication unit configured to receive information on at least one component included in the object, state information on the at least one component, and augmented reality information on the component in which an event has occurred;
an augmented reality processing unit generating an augmented reality interface based on the augmented reality information;
a display unit for displaying the augmented reality interface by overlapping the captured image; and
A control unit for determining whether the event has occurred for each of the at least one component based on the received state information,
The control unit is
comparing the state information of a specific component among the at least one component with a predetermined normal operation reference value to determine whether the specific component operates normally;
When it is determined that the specific component is not operating normally, the cause of the abnormal operation is determined based on the state information, and an event including the cause of the abnormal operation is generated,
Selecting a component in which the event occurred, and requesting the augmented reality information for the selected component
Augmented reality providing system.
delete 11. The method of claim 10,
The controller may request the augmented reality information for generating different augmented reality interfaces according to the abnormal operation included in the event.
11. The method of claim 10,
The augmented reality processing unit generates different augmented reality interfaces according to the cause of the abnormal operation included in the event,
The display unit, an augmented reality providing system for overlapping the generated augmented reality interface with the component in which the event occurred.
11. The method of claim 10,
a user terminal including the camera unit, the communication unit, the display unit, and the control unit;
Containing a server including the object recognition unit and the augmented reality processing unit,
Augmented reality providing system.

Images