KR20190081177A - Device and method for providing augmented reality user interface - Google Patents

Abstract

The present invention relates to a system and method for providing augmented reality. According to the present invention, the method for providing augmented reality comprises the following steps: (a) recording an image including an object; (b) recognizing the object included in the recorded image; (c) receiving information on a plurality of components included in the object; (d) dividing an area corresponding to each of the plurality of components from the entire area of the recorded object based on the information; (e) generating an augmented reality interface including an augmented reality representative icon for each of the plurality of components; and (f) displaying each generated augmented reality interface on the corresponding divided area.

Description

Technical Field [0001] The present invention relates to augmented reality providing system and method,

The present invention relates to augmented reality providing system and method. Specifically, the present invention relates to an augmented reality providing system and method for displaying an augmented reality representative icon of a component included in a device being photographed in corresponding areas.

Augmented reality (AR) is a computer graphics technique that combines virtual objects or information into a real environment to make them look like objects in the original environment.

The augmented reality has a characteristic that it can reinforce and provide additional information that is difficult to obtain only in the real world by synthesizing a virtual object on the basis of the real world, unlike the existing virtual reality which only deals with a virtual space and objects. Because of this feature, the augmented reality can be applied to various real environments unlike existing virtual reality, which can be applied only to fields such as games. Especially, it is attracting attention as next generation display technology suitable for ubiquitous environment.

Research on augmented reality technology has been continued since the start of research on augmented reality technology. Recently, augmented reality technology has been widely used in location based services with the proliferation of smart phones.

Generally, the augmented reality technology displays information about the device currently being photographed on a preview screen in a menu form. At this time, the preview screen can display the status information of the device being photographed in 2D or 3D form.

The device includes various components. At this time, each component may be displayed as a separate augmented reality icon. However, when the augmented reality information on the components included in the device is provided to the user all at once, the user interface (UI) becomes complicated and the user has difficulty in grasping the desired information properly.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a system and method for providing an augmented reality capable of simplifying a user interface (UI) to an augmented reality by expressing concrete information about the components included in the apparatus being photographed in augmented reality submenu .

In addition, the present invention determines whether or not an abnormal operation is necessary for a plurality of components included in a device being photographed, and notifies the user of an abnormal operation and an advertisement for the replaceable model to an augmented reality submenu of the corresponding component It is an object of the present invention to provide a system and method for providing an augmented reality that enables a user to select and browse only desired information.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention which are not mentioned can be understood by the following description and more clearly understood by the embodiments of the present invention. It will also be readily apparent that the objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

According to another aspect of the present invention, there is provided an augmented reality providing method including: (a) capturing an image including an object; (b) recognizing an object included in the captured image; (C) receiving information about a plurality of components, (d) identifying an area corresponding to each of the plurality of components on the entire area of the photographed object based on the information, (E) generating an augmented reality interface including augmented reality representative icon for each of the elements, and displaying each of the augmented reality interfaces generated on the corresponding divided area.

The method may further include receiving a selection of the component from a user, and developing an augmented reality submenu for the selected component.

The step (e) of generating the augmented reality interface may further include receiving status information on the plurality of components, comparing the status information of the components with a predetermined normal operation reference value, Determining whether an abnormal operation of the component is required or not, and calculating the remaining service life based on the status information of the component to determine necessity of replacement of the component.

In addition, the step (e) of generating the augmented reality interface may include: generating a notification about the abnormal operation when an abnormal operation occurs in the component; and if the component is in need of the replacement, Generating an advertisement for the replaceable model of the component, and assigning the generated alert or the advertisement to the augmented reality submenu of the component.

The step (f) of displaying the augmented reality interface on the corresponding divided area may include expanding the augmented reality submenu including the notification or the advertisement only when the user selects the corresponding element .

The step of generating the notification of the abnormal operation may include generating an augmented reality interface including the augmented reality representative icons of different shapes or colors according to the cause of the abnormal operation.

The step (f) of displaying the augmented reality interface on the corresponding divided area may include displaying the augmented reality image for the replaceable model of the component included in the advertisement on the component in an overlapping manner can do.

In addition, the step of determining necessity of replacement of the component may include calculating the remaining service life by comparing the use period of the component with the recommended service life of the component, Lt; RTI ID = 0.0 > lifetime. ≪ / RTI >

The step (b) or (e) may be performed in the server, and the remaining steps may be performed in the user terminal.

According to another aspect of the present invention, there is provided an augmented reality providing system including a camera unit for capturing an image including an object, an object recognizing unit for recognizing the object included in the image, , A communication unit for receiving the augmented reality information or the augmented reality interface for the plurality of components, the augmented reality representative icon for each of the plurality of components based on the received state information and the augmented reality information, An augmented reality processing unit for generating an augmented reality interface including the augmented reality interface, a display unit for displaying the augmented reality interface, and an area corresponding to each of the plurality of components on the object, And a controller for displaying the representative interface on the corresponding divided area. The.

The apparatus may further include an interface unit receiving an input of a user selecting one of the plurality of components, and the controller may control the augmented reality interface so that the augmented reality submenu for the selected component is developed have.

The controller may compare the state information of the component with a predetermined normal operation reference value to determine whether the specific component is abnormal or not, and when an abnormal operation occurs in the component, May be generated and allocated to the augmented reality submenu.

In addition, the control unit may calculate the remaining lifetime of the component to determine necessity of replacement, and if there is a necessity of the replacement, generate an advertisement for the replaceable model of the component, . ≪ / RTI >

In addition, the augmented reality interface may include augmented reality representative icons of different shapes or colors depending on the type of the augmented reality submenu.

The mobile terminal may further include a user terminal including the camera unit, the communication unit, the display unit, and the controller, and a server including the object recognition unit and the augmented reality processing unit.

In the system and method for providing an augmented reality according to the present invention, a constituent element included in a device being photographed is represented by an augmented reality representative icon, and concrete information about each constituent element is represented by a menu of augmented reality. This allows the user to intuitively and easily grasp information about each component included in the device.

In addition, the system and method for providing an augmented reality according to the present invention may further include a step of determining whether or not each of the plurality of components included in the device being photographed is abnormal or not and detecting an augmented reality interface having different shapes and colors to provide. Accordingly, the present invention can shorten the time required for device management, thereby enhancing the user's convenience and improving the processing speed. In addition, since a quick work process can be performed within the same time, the working efficiency can be improved.

In addition, the augmented reality providing system and method of the present invention notifies a component requiring replacement, and provides an advertisement for a replaceable model to a user. Therefore, the user can easily obtain the information on the substitutable model without any search, and the seller can use it as a marketing and promoting means. Thus, the present invention can satisfy the needs of the user and the needs of the seller at the same time.

The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

1 is a diagram illustrating an augmented reality providing system according to an embodiment of the present invention.
2 is a block diagram for explaining an augmented reality providing apparatus according to an embodiment of the present invention.
3 is a diagram illustrating an operation method of an augmented reality providing apparatus according to an embodiment of the present invention.
4 is a flowchart illustrating a method of providing an augmented reality according to an embodiment of the present invention.
FIG. 5 is a flowchart for illustrating the step of displaying the augmented reality interface of FIG.
6 is a flowchart for explaining the step of determining the presence or absence of an abnormal operation in FIG.
FIG. 7 is a flowchart for explaining the step of determining necessity of replacement in FIG.
8 to 10 are views for explaining the operation of the augmented reality providing method according to an embodiment of the present invention.

The above and other objects, features, and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings, which are not intended to limit the scope of the present invention. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to denote the same or similar elements.

Hereinafter, a system and method for providing an augmented reality according to the present invention will be described with reference to Figs. 1 to 10. Fig.

1 is a diagram illustrating an augmented reality providing system according to an embodiment of the present invention.

Referring to FIG. 1, an augmented reality providing system according to an embodiment of the present invention includes a user terminal 100, a target device 200, and a server 300.

The user terminal 100 may operate as an augmented reality providing apparatus. The user terminal 100 photographs the target device 200 and receives information about at least one component included in the target device 200 from the server 300. [

Then, the user terminal 100 receives status information on at least one component included in the target device 200, and determines whether there is an abnormal operation for each of the at least one component based on the received status information, .

Then, the user terminal 100 receives the augmented reality information for each component from the server 300, generates an augmented reality interface based on the augmented reality information, and displays the augmented reality interface on the screen. Here, the augmented reality information includes the augmented reality-related data of the constituent elements included in the photographed target device 200 for creating the augmented reality interface. The augmented reality information can be modeled and rendered as a 2D or 3D image, and the rendered image is converted into an augmented reality interface and displayed in a preview form on the screen of the user terminal 100. [

At this time, the augmented reality interface includes an augmented reality submenu including an announcement about an abnormal operation and an advertisement about necessity of replacement. The augmented reality submenu can be deployed only when the user selects the corresponding element. That is, the user can select and browse the advertisement of the desired component or the detailed information of the advertisement.

Before the user selects the component, the component may be displayed on the designated area of the device with the augmented reality representative icon. At this time, the augmented reality representative icon may be represented in different shapes or colors depending on the state of the corresponding component. A detailed description thereof will be described later.

The user terminal 100 may be a mobile-based device, a wearable device (e.g., a head mounted device (HMD), a video see-through device, An optical see-through device), a screen-based device, and a projector-based device.

Also, the user terminal 100 may be a computer, a UMPC (Ultra Mobile PC), a workstation, a netbook, a PDA (Personal Digital Assistants), a portable computer, a web tablet, a wireless mobile phone, a mobile phone, a smart phone, an e-book, a portable multimedia player (PMP), and the like. However, the present invention is not limited thereto.

The target device 200 may be various electronic devices including at least one component. For example, the target device 200 may be an automatic controller, industrial device, or switchboard. At this time, the target device 200 may include a circuit breaker, a relay, a communication link device, a watt-hour meter, a sensor unit, an HMI (Human Machine Interface), and the like. The description of each component is already known, and a detailed description thereof will be omitted here.

Also, the target device 200 can be an object of measurement of an Automatic Meter Reading (AMR). Generally, the AMR refers to converting measurement results from an object at a remote location to an electrical signal, transmitting the data through the communication network, and processing the data in the computer.

The server of the AMR system can receive a plurality of data (e.g., data) through a data communication network (e.g., a backbone network, a private network, and an Internet network) provided to perform data communication based on a Transmission Control Protocol / Internet Protocol Collecting device. The server transmits and receives data in real time to and from the data collection device through the data communication network, and it is possible to monitor the power status of each measurement object in real time.

In addition, the server receives meter reading data of each measurement object from the data collection device in real time. Based on such meter reading data, the power state of each measurement target, for example, power failure, electrical conduction, leakage current state, etc., is analyzed and provided to the manager.

Here, the meter reading data may be the sum of the used electric power amounts of all the measurement objects allocated to one data centralizing apparatus, the used electric power amount per each measurement subject, the instantaneous electric power amount, and the like, and all the data May be included.

The server 300 includes a database for storing various kinds of information about the target device 200. For example, the server 300 may include object information about the target device 200, information about the components included in the target device 200, status information about the respective components, Can be stored.

The server 300 may operate as a data collection device or server of an AMR system. That is, the server 300 can receive and store the meter reading data for the target device 200 in real time. In this way, the server 300 can monitor and remotely control the operation status of the target device 200. However, the present invention is not limited thereto.

2 is a block diagram for explaining an augmented reality providing apparatus according to an embodiment of the present invention.

2, a user terminal 100 as an augmented reality providing apparatus according to an exemplary embodiment of the present invention includes a control unit 110, a camera unit 120, an object recognition unit 130, a position measurement unit 140, An augmented reality processing unit 150, a communication unit 160, a sensor unit 165, a memory unit 170, a display unit 180, and an interface unit 190. However, the components of the user terminal 100 shown in FIG. 2 are not essential, so that the user terminal 100 having more or fewer components can be implemented.

The control unit 110 may control the overall operation of the components included in the user terminal 100. The control unit 110 may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the control unit 350 can access, store, manipulate, process, and generate data in response to execution of the software. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media. The detailed operation of the control unit 110 will be described below.

The camera unit 120 is a device for capturing an image, and the captured image is transmitted to the control unit 110. The image photographed by the camera unit 120 may be displayed on the display unit 180 in a real-time preview format.

The object recognition unit 130 recognizes an object included in the image photographed by the camera unit 120. The object recognition method used in the object recognition unit 130 includes a marker based recognition method for recognizing an object using a marker included in an image and a markerless recognition method for recognizing an actual object registered in advance instead of a physical marker.

Specifically, the marker-based recognition method is a method of recognizing an object based on a marker such as a QR code, a data matrix (DataMatrix), or a 2D barcode. Since the marker-based recognition method utilizes a special type of marker that the system understands in advance, the operation speed and the recognition rate are high.

The markerless recognition method is divided into a natural image-based method and an environmental recognition-based method. The natural image-based method analyzes a large number of feature points existing in a general plane image, and uses the analyzed feature points as invisible markers. The natural image-based method uses a technique of extracting a feature point existing in a natural image and a technique of effectively converting the feature point into a descriptor for describing the feature point. The environment recognition based method is a technique of recognizing the environment itself as a three-dimensional map without a special type of marker or a pre-input image. The environment recognition based method can be a mapping technique for realizing a three-dimensional environment map by analyzing data from a camera or an RGBD sensor and a camera tracking technique for estimating the position of the current camera on the created map have.

The object recognition unit 130 can recognize the objects included in the photographed image by using the marker-based recognition method and the markerless recognition method described above. At this time, the object recognition unit 130 can recognize the object included in the image based on the position information on which the image was captured, the marker for the object, or the markerless information on the object. However, the method of recognizing the object by the object recognizing unit 130 is not limited to this, and various newly developed methods can be applied.

However, the operation of the object recognition unit 130 may be performed by a separate external device (e.g., the server 300), and only the result data on the execution result may be received and used by the user terminal 100 .

The position measuring unit 140 measures the current position of the user terminal 100 and the position of the object to be photographed. In more detail, the position measuring unit 140 may measure the current position of the user terminal 100 and the position of the object photographed by the user terminal 100 using the local position tracking technique. Local location tracking techniques may be used in accordance with the manner in which they are used, such as time of flight (TOF), spatial scan, inertial sensing, mechanical linkage, phase difference sensing, (direct field sensing), and the above techniques may be used in combination to increase the accuracy.

The location measurement unit 140 can perform location measurement using a location based service using near field communication (NFC). For example, the location based service may use Zigbee, iBeacon technology. The location based service can be used mainly in indoor environment where GPS can not be used and the relative position can be grasped through the signal strength of each node. In addition, the position measuring unit 140 can increase the accuracy of position measurement using various sensors (inertial measurement sensor, altimeter, compass) mounted on the user terminal 100. [

The augmented reality processing unit 150 may generate the augmented reality interface provided to the user based on the augmented reality information. The augmented reality information is raw data used for modeling and rendering objects to be photographed. The augmented reality information may be stored in the database of the server 300 or may be stored in the memory unit 170 of the user terminal 100 and used.

The augmented reality processing unit 150 can use a modeling technique for accurately expressing an actual object virtually on the basis of the augmented reality information and a rendering technique for realistically expressing it. The modeling technique is a technique of synthesizing the 2D image input by the video by reflecting the depth information. The modeling technique enables various applications that can not be achieved in the real space by performing object transformation and color change according to the user's intention. Rendering technology is a technology that enables the user to feel the vividness in the augmented reality interface by realistically expressing the virtual model by adding lighting effect, shadow effect, brightness change, etc. to the created virtual models.

However, the operation of the augmented reality processing unit 150 may be performed by a separate external device (for example, the server 300), and only the result data on the execution result may be received and used by the user terminal 100 .

The communication unit 160 exchanges data with the server 300 that stores and manages the augmented reality database or the target device 200 to be imaged. At this time, the augmented reality database is a database storing attribute information of augmented reality objects and augmented reality objects.

The communication unit 160 may include an electronic device, a module, or a program that exchanges data with other terminal devices by wire or wireless. The communication protocol used by the communication unit 160 includes, for example, CDMA (Code Division Multiple Access) communication, WCDMA (Wideband Code Division Multiple Access) communication, or broadband wireless communication. However, the present invention is not limited thereto.

The sensor unit 165 is a device for detecting information required for the augmented reality or context information that is further applied to the augmented reality. The sensor unit 165 may include a temperature sensor, a humidity sensor, a position sensor, a motion sensor, and the like. Here, a GPS sensor for sensing a GPS signal may be used as the position detection sensor, and a gyroscope sensor or an acceleration sensor may be used as the motion detection sensor. In addition, the sensor unit 165 may measure the current position, the movement, the tilt, the direction of the user terminal 100, the distance to the object to be photographed, the surrounding sound, brightness, and the like.

The memory unit 170 stores an operating system, an application program, and storage data (telephone number, SMS message, compressed image file, moving image, etc.) for controlling the overall operation of the user terminal 100. Also, the memory unit 170 may store the augmented reality object corresponding to the object to be imaged, the attribute information of the augmented reality object, and the state information. That is, the memory unit 170 stores the object to be imaged and the state information and the augmented reality information about the components included in the object.

The memory unit 170 may include a random access memory (RAM) or a read only memory (ROM) and a static storage device for storing static information and instructions. The memory unit 170 may be a volatile memory unit or a non-volatile memory unit. The memory portion 170 may also be another type of computer readable medium, such as a magnetic or optical disk.

The display unit 180 displays status information (or an indicator) generated during operation of the user terminal 100, limited numbers and characters, a moving picture, and a still picture. Also, the display unit 180 may display an image received through the camera unit 120 and display an augmented reality interface corresponding to the object to be photographed.

The interface unit 190 can receive input from the user. The interface unit 190 may include an input device such as a button-type input unit, a touch screen, a keyboard, a mouse, or the like. For example, the touch screen may be formed on the upper surface of the display unit 180 and receive the input through the user's touch. However, the present invention is not limited thereto.

3 is a diagram illustrating an operation method of an augmented reality providing apparatus according to an embodiment of the present invention.

3, a user terminal 100, which is an augmented reality providing apparatus according to an embodiment of the present invention, photographs a target device 200 and displays an augmented reality interface for the photographed target device 200 on a display unit 180).

Here, the target device 200 may include first to third devices 210, 220 and 230, and each device may include a plurality of components. The target device 200 may be an automatic controller, an industrial device, or an electric switchboard.

For example, the first device 210 may include a plurality of components A1, B1, C1, and D1. At this time, the plurality of components A1, B1, C1, and D1 may be any one of a circuit breaker, a relay, a communication link device, a watt-hour meter, a sensor unit, and an HMI. However, this is merely one example, and the present invention is not limited thereto.

The server 300 interworks with the target device 200 to store various data for the target device 200. [ The server 300 includes an object information DB 310, a component DB 320, a status information DB 330, and an augmented reality DB 340. Hereinafter, the target device 200 will be described in combination with the term " object "

The object information DB 310 includes various information about the object. For example, it may include information such as product information (product name, product category, installation time, management method, etc.) of the object, location information where the object is installed, circuit diagram and installation diagram related to the object.

The component DB 320 includes information on all components included in the object. The component DB 320 includes information on components included in each object, a connection relationship between the components, a normal operation range of each predetermined component, and a replaceable component in the event of a failure.

For example, in the case of a breaker included in a distributor, it includes a VCB (Vacuum Circuit Breaker), an ACB (Air Circuit Breaker), an MCCB (Molded Case Circuit Breaker) (320) may include information about the type of breaker included in the target distributor and the replaceable breaker.

The state information DB 330 includes operation information of each component included in the object. For example, the state information DB 330 may include power amount information, temperature information, voltage information, or current information for each object component. The status information of each component can be updated in real time and can be used as part of an AMR.

The augmented reality DB 340 stores augmented reality information, which is raw data used for modeling and rendering an object. Also, the augmented reality DB 340 stores the augmented reality interface provided to the user based on the augmented reality information. That is, the augmented reality DB 340 stores both the augmented reality information before modeling and rendering of an object, the augmented reality interface after modeling and rendering, and provides necessary data according to a request of the user terminal 100 .

The user terminal 100 recognizes the target device 200 that is an object included in the photographed image and receives the augmented reality information or the augmented reality interface for the components included in the target device 200 from the server 300 Display it on the screen.

For example, the augmented reality representative icon for the plurality of components (A1, B1, C1, D1) of the first device 210 that are not actually visible in the augmented reality interface) is overlapped on the first device 210 Can be expressed. At this time, the augmented reality representative icon may be expressed in different shapes or colors depending on whether the components A1, B1, C1, and D1 are abnormal or need to be replaced.

Also, the augmented reality submenu including the advertisement for the replaceable model or the advertisement about the abnormal operation for each of the components A1, B1, C1, and D1 may be allocated to the augmented reality representative icon. At this time, the user can expand the menu of the augmented reality allocated to the selected component by selecting a specific component, and can view detailed information about the advertisement or advertisement.

A detailed description of the augmented reality providing method performed by the user terminal 100 will be described later.

4 is a flowchart illustrating a method of providing an augmented reality according to an embodiment of the present invention.

Referring to FIG. 4, an augmented reality providing method according to an embodiment of the present invention first captures an image including an object in a camera unit 120 of a user terminal 100 (S110).

Then, the object recognition unit 130 recognizes an object included in the photographed image (S120).

Then, the control unit 110 receives information on at least one component included in the object through the communication unit 160 (S130). The control unit 110 can receive from the server 300 the information about the recognized object and the information about the elements included in the recognized object. In addition, when information on the object has been received in the past, the controller 110 may receive information on the object from the memory unit 170 that stores the object information. At this time, the number of components included in the object may be plural.

Then, the control unit 110 receives status information about at least one component included in the object from the server 300 through the communication unit 160 (S140). The server 300 stores the status information of each component included in the object in real time based on the AMR. The controller 110 may request and receive current state information of each component stored in the server 300. [

Then, the controller 110 generates an augmented reality interface including the augmented reality representative icon for each component included in the object based on the received state information and the information about the component, and displays the generated augmented reality interface on the display unit 180 (S150). Herein, the augmented reality interface includes an augmented reality representative icon representing a component and an augmented reality submenu including details of the state information of the component.

At this time, the control unit 110 can generate notification of whether or not an abnormal operation of a plurality of components included in the object is performed based on the received state information, and an advertisement for a replaceable model according to necessity of replacement. The generated notification or advertisement can be assigned to the augmented reality submenu of the corresponding component.

In addition, an augmented reality representative icon for each of a plurality of components can be displayed on an area corresponding to each component. The areas of the plurality of components can be classified and allocated based on the information about each component received from the server 300.

Then, the controller 110 can receive the augmented reality information of each component from the server 300. [ The augmented reality processing unit 150 generates an augmented reality interface based on the received augmented reality information and the augmented reality submenu allocated to each component, and displays the generated augmented reality interface on the display unit 180. [

Then, the control unit 110 receives a selection of a user's component through the interface unit 190 (S160).

Next, the controller 110 controls the augmented reality interface to develop the augmented reality submenu for the selected component (S170).

That is, in the method of providing an augmented reality according to the present invention, the augmented reality interface is configured such that a constituent element included in a device being photographed is represented by an augmented reality representative icon, and specific information about each constituent element is represented by augmented reality submenu.

This allows the user to intuitively and easily grasp information about each component included in the device. In addition, the user can select and inquire detailed information about the desired component, thereby improving the user's convenience.

FIG. 5 is a flowchart for illustrating the step of displaying the augmented reality interface of FIG.

5, displaying the augmented reality interface including the augmented reality representative icon for each component included in the object of step S150 in Fig. 4 may be performed in the following order in the controller 110 . Hereinafter, the contents overlapping with those described with reference to FIG. 4 will be omitted and the differences will be mainly described.

First, the control unit 110 divides the area corresponding to each component (S251). The control unit 110 can distinguish areas corresponding to the respective components on the photographed object based on the information received from the component DB 320 of the server 300. [

For example, the control unit 110 identifies the entire area of the object by recognizing the outline (or corner) of the photographed object. Then, the controller 110 receives the location information of each element of the object from the server 300. Then, the control unit 110 identifies the region corresponding to each component based on the entire region of the object, and matches the region corresponding to each component by matching each component to the region. However, this is only an example, and the present invention is not limited thereto.

Then, the controller 110 displays the augmented reality image for the corresponding element on the divided area (S252). At this time, the augmented reality image is displayed as an augmented reality representative icon representing the corresponding element, and can be displayed so as to overlap on the corresponding divided area. The augmented reality image may be generated based on the augmented reality information received from the server 300. [

Next, the control unit 110 determines whether there is an abnormal operation of each component (S253). The controller 110 can determine the presence or absence of an abnormal operation based on the status information of each component. A detailed description of determining whether or not an abnormal operation is to be performed will be described later with reference to Fig.

Then, the control unit 110 determines whether or not each component is required to be replaced (S254). The controller 110 may calculate the remaining service life of each component by using the life determination algorithm and determine the necessity of replacement of each component. A detailed description of the life judgment algorithm will be given later with reference to Fig.

Subsequently, when an abnormal operation is found in each component, the control unit 110 generates an alert for the abnormal operation. If the necessity of replacing each component is recognized, the control unit 110 generates an advertisement for the replaceable model (S255).

Then, the controller 110 allocates the created notification or advertisement to the augmented reality submenu (S256). At this time, the augmented reality interface can display the shape and color of the augmented reality image (that is, the augmented reality representative icon) differently depending on the presence or absence of the advertisement or advertisement.

Then, when the user selects a specific component, the controller 110 develops the augmented reality submenu allocated to the selected component (S270).

However, the operation of determining whether there is an abnormal operation in step S253 described above, the operation of determining necessity of replacement of the component in step S254, the notification of the abnormal operation in step S255, and the operation of generating an advertisement for the replaceable model May be selectively implemented. That is, in some embodiments of the present invention, some or all of steps S253 to S255 may be omitted and implemented. However, this is merely one embodiment, and the present invention is not limited thereto.

6 is a flowchart for explaining the step of determining the presence or absence of an abnormal operation in FIG.

Referring to FIG. 6, in step S253 of FIG. 5, the control unit 110 may determine whether there is an abnormal operation for each of the components included in the object in the following order.

First, the state information of a specific component is compared with a predetermined normal operation reference value to determine whether the specified component is operating normally (S351, S353). For example, the status information may include power information, temperature information, voltage information, or current information for the component. The normal operation reference value can be previously stored in the memory unit 170 and used as a normal operation range for each state numerical value.

Next, when it is determined that the specified component is not in normal operation, the controller 110 determines the cause of the abnormal operation based on the state information of the specified component (S355). For example, if the temperature of the specified component deviates from the normal operating range, it is determined that the temperature abnormality occurs, and if the power amount of the specified component deviates from the normal operation range, it can be determined that the power amount is abnormal.

Subsequently, the control unit 110 generates a notification including the cause of the determined abnormal operation (S357). The information about the corresponding component in which the notification is generated may be transmitted to the augmented reality processing unit 150 or may be stored in the memory unit 170. [

Next, although not explicitly shown in the drawing, the control unit 110 repeats steps S351 to S357 for the other components. Thus, it is possible to judge whether all the components included in the object operate normally or not.

In addition, the augmented reality processing unit 150 may generate different augmented reality representative icons for the respective components in accordance with the cause of the abnormal operation included in the notification. At this time, the augmented reality representative icon can be generated using different modeling and rendering images depending on the cause of the abnormal operation. In addition, the augmented reality representative icons can be expressed in different colors according to the degree of abnormal operation.

For example, an augmented reality representative icon (for example, a fire rendering image) generated when the cause of the abnormal operation is a temperature and an augmented reality representative icon (for example, a lightning rendering image) May be different. In addition, when the temperature of the component in which the abnormality is generated is 100 degrees and 150 degrees, the colors can be expressed in different colors. However, this is only an example, and the present invention is not limited thereto.

Then, the control unit 110 displays the augmented reality representative icon generated in the augmented reality processing unit 150 in the divided area of the corresponding component in which the notification is generated.

In addition, in another embodiment of the present invention, instead of receiving the augmented reality information on the cause of the abnormal operation, the controller 110 may provide an augmented reality interface including the augmented reality representative icon previously stored in the server 300 for the corresponding element And display it on the display unit 180 immediately.

For example, the server 300 may perform the operations of the object recognition unit 130 and the augmented reality processing unit 150 of the user terminal 100 described above. Then, the user terminal 100 can receive and use the resultant data of the operation performed from the server 300. [

Specifically, the user terminal 100 transmits the image captured by the camera unit 120 to the server 300, and the server 300 recognizes the object included in the received image. Then, the server 300 may generate an augmented reality interface including an augmented reality representative icon for each component included in the recognized object, and provide the generated augmented reality interface to the user terminal 100.

In this case, the user terminal 100 directly outputs the data received from the server 300 (i.e., the augmented reality interface) to the display unit 180. [ Therefore, the operation amount of the control unit 110 is reduced, so that the operation speed can be increased.

.

FIG. 7 is a flowchart for explaining the step of determining necessity of replacement in FIG.

Referring to FIG. 7, in step S254 of FIG. 5, the lifetime determination algorithm for determining necessity of replacement of each component included in the object may be performed by the controller 110 in the following order.

First, the life judgment algorithm determines whether the use period of the corresponding component is within the recommended service life (S451). The life estimation algorithm can calculate the recommended life span based on the statistical data on the average life span of products of the same type as those of the component by comparing the manufacture date of the component with the current date to calculate the service life span.

Then, the life determination algorithm determines whether the corresponding component is operating normally based on the state information of the component (S453). Specifically, the life judgment algorithm can determine whether the corresponding component is operating normally by considering the operation state of each component included in the component.

Subsequently, the life estimation algorithm predicts the remaining service life of the photographed component based on the component information and the state information (S455).

Specifically, the life estimation algorithm can calculate the remaining lifetime of the photographed component by comparing the lifetime of the component with the recommended lifetime of the component. In addition, the life estimation algorithm can calculate the remaining service life of the corresponding component in consideration of the wear state, fusion degree, fuse state, bypass switch operation state, or breaker operation state of each component included in the component .

Then, the life determination algorithm determines necessity of replacing the photographed component (S457).

For example, if the lifetime of the photographed component exceeds the recommended lifetime, the lifetime determination algorithm determines that the component needs to be replaced. Also, when the remaining lifetime of the component is smaller than a preset reference value, it can be determined that there is a necessity of replacing the corresponding component. Here, the predetermined reference value at this time may be a period necessary for replacing the component (for example, a construction period, a component order period). However, this is an example of the present invention, and the present invention is not limited thereto.

8 to 10 are views for explaining the operation of the augmented reality providing method according to an embodiment of the present invention.

8, in the method of providing an augmented reality according to an exemplary embodiment of the present invention, a user terminal 100 photographs a target device 210, and a target device 210 includes first through fourth components A1 , B1, C1, D1). The user terminal 100 may display the augmented reality interfaces corresponding to the first through fourth components A1, B1, C1 and D1 on the target device 210 in an overlapping manner. The augmented reality interface includes an augmented reality representative icon representing a component and an augmented reality submenu including details of the state information of the component.

The controller 110 may divide the regions corresponding to the respective elements based on the information about the respective elements, and may display the augmented reality representative icons representing the respective elements in the divided regions.

The control unit 110 determines whether or not abnormal operation of the first to fourth components A1, B1, C1, and D1 based on the state information of the first to fourth components A1, B1, C1, The necessity of replacement can be judged.

If an abnormal operation occurs in a specific component, the controller 110 generates a notification of the abnormal operation. In addition, when it is determined that there is a need to replace the specific component, the control unit 110 generates an advertisement for the replaceable model. At this time, the display (M11) for the generated advertisement and the display (M21) for the advertisement may notify the corresponding component that the advertisement or advertisement has occurred, but may not include the detailed information. At this time, the display (M11) for the generated notification and the display (M21) for the advertisement can be displayed only when the user selects (for example, clicks or touches) the corresponding component.

That is, even if an abnormal operation occurs in a specific component or it is judged that there is a necessity of replacement, only when the user selects the component, the display M11 for the generated notification and the display M21 for the advertisement A submenu can be displayed.

At this time, the presence or absence of the advertisement or advertisement of the corresponding component can be expressed by the shape or color of the augmented reality representative icon of the corresponding component. For example, although not clearly shown in the drawing, if the cause of the abnormal operation is temperature, the augmented reality representative icon may include a flame rendering image. If the abnormal operation cause is the amount of power, the augmented reality representative icon includes a lightning rendering image . At this time, the shape or color of each icon can be expressed by an animation effect (for example, flicker, color change, left / right vibration, etc.). However, this is merely one example, and the present invention is not limited thereto.

The user can select the component that he wants to view (for example, the B1 component in which the notification has occurred or the C1 component in which the advertisement has occurred), and inquire the information contained in the augmented reality submenu for that component . For example, the user can select a component through which the advertisement or advertisement is generated through the interface unit 190 including the touch screen.

Referring to FIG. 9, the user can select a 'notification' display of the second component B1 included in the target device 210 through the interface unit 190 of the user terminal 100. FIG.

In this case, the controller 110 may display an augmented reality submenu M12 including concrete information on 'notification' of the selected component B1. Herein, the augmented reality submenu M12 may include detailed information related to the cause of the anomaly.

For example, when the cause of the abnormality is the temperature, the augmented reality processing unit 150 develops the augmented reality submenu including the normal temperature range of the constituent element, the temperature of the current constituent element, and the cooldown method of the constituent element can do.

As another example, if the cause of the abnormality is the power amount, the augmented reality processing unit 150 includes information on the range of the normal power amount of the component, the current power amount, a graph indicating the change in the amount of power from the past to the present It is possible to develop an augmented reality submenu. However, this is merely one example, and the present invention is not limited thereto.

Therefore, the apparatus for providing an augmented reality of the present invention determines whether or not each of the plurality of constituent elements included in the apparatus being photographed is abnormal or not, and provides the augmented reality interface of different shapes and colors according to the cause of the abnormal operation . Accordingly, the present invention can shorten the time required for device management, thereby enhancing the user's convenience and improving the processing speed. In addition, since a quick work process can be performed within the same time, the working efficiency can be improved.

Referring to FIG. 10, the user can select 'advertisement' display of the third component C1 included in the target device 210 through the interface unit 190 of the user terminal 100.

In this case, the controller 110 may display an augmented reality submenu M22 including concrete information on 'advertisement' of the selected component C1. Here, the augmented reality submenu M22 may include a manufacturing date, an expected remaining lifetime, state information, and replaceable model information. However, this is merely one example, and the present invention is not limited thereto.

At this time, if there are a plurality of replaceable models, the controller 110 may display a list of replaceable models arranged on the basis of the compatibility with the object or the month of manufacture, on the augmented reality submenu M22. The controller 110 can display the list of the most recently manufactured products in the order of the highest compatibility with the objects in the augmented reality submenu M22. However, the sorting order is not limited to the above-described order, and the sorting order may be arranged to be provided to the user through a preset reference.

For example, the A to C models may be displayed in the augmented reality submenu M22 in the order of high compatibility with the selected component C1. If the user selects model A, the augmented reality image for model A may overlap and be displayed on the selected component C1. Through this, the user can intuitively recognize through the augmented reality the appearance when the replaceable model of the component is actually installed.

Therefore, the method of providing an augmented reality of the present invention notifies a component requiring replacement, and provides an advertisement for a replaceable model to a user. Through this, users can easily obtain information on the substitutable models without any search, and sellers can use them as marketing and promotional means. Thus, the present invention can satisfy the needs of the user and the needs of the seller at the same time.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

100: User terminal
200: Target device
300: server

Claims (15)

A method for providing an augmented reality performed by a user terminal or a server,
(A) photographing an image including an object;
(B) recognizing an object included in the photographed image;
(C) receiving information about a plurality of components included in the object;
(D) identifying an area corresponding to each of the plurality of components on the entire area of the photographed object based on the information;
(E) generating an augmented reality interface including an augmented reality representative icon for each of the plurality of components; And
And displaying each of the generated augmented reality interfaces on the corresponding divided area (f)
Method of providing augmented reality.
The method according to claim 1,
Receiving a selection from the user for the component;
Further comprising: developing an augmented reality submenu for the selected component.
The method according to claim 1,
The step (e) of generating the augmented reality interface includes:
Receiving status information for the plurality of components;
Comparing the state information of the component with a predetermined normal operation reference value to determine whether the component is abnormal;
Calculating a remaining lifetime based on the state information of the component, and determining necessity of replacement for the component.
The method of claim 3,
The step (e) of generating the augmented reality interface includes:
Generating an alert for the abnormal operation when an abnormal operation occurs in the component;
Generating an advertisement for the replaceable model of the component if the component has the need to replace;
And assigning the generated notification or the advertisement to an augmented reality submenu of the component.
5. The method of claim 4,
(F) displaying the augmented reality interface on the corresponding divided area,
And expanding the augmented reality submenu including the advertisement or the advertisement only when the user selects the corresponding element.
5. The method of claim 4,
Wherein the step of generating the notification of the abnormal operation includes generating an augmented reality interface including the augmented reality representative icons of different shapes or colors in accordance with the cause of the abnormal operation.
5. The method of claim 4,
(F) displaying the augmented reality interface on the corresponding divided area,
And displaying an augmented reality image for a replaceable model of the component included in the advertisement by overlapping the augmented reality image with the component.
The method of claim 3,
Wherein the step of determining necessity of replacement for the component comprises:
Calculating a remaining service life of the component by comparing the service life of the component with a recommended service life of the component,
And calculating the remaining lifetime based on the state information of the component.
The method according to claim 1,
Wherein the step (b) or (e) is performed in the server, and the remaining step is performed in the user terminal.
A camera unit for capturing an image including an object;
An object recognition unit for recognizing the object included in the image;
A communication unit for receiving status information on a plurality of components included in the object and augmented reality information or augmented reality interface for the plurality of components;
An augmented reality processing unit for generating an augmented reality interface including augmented reality representative icon for each of the plurality of components based on the received state information and the augmented reality information;
A display unit displaying the augmented reality interface; And
And a control unit for distinguishing regions corresponding to the plurality of components on the object and displaying the augmented reality representative interface for each component on the corresponding divided region
Augmented reality providing system.
11. The method of claim 10,
Further comprising an interface unit for receiving an input of a user selecting any one of the plurality of components,
Wherein the controller controls the augmented reality interface so that the augmented reality submenu for the selected component is developed.
12. The method of claim 11,
Wherein,
Comparing the state information of the component with a predetermined normal operation reference value to determine whether the component is abnormal or not,
And generates an alert for the abnormal operation and assigns the alert to the augmented reality submenu when an abnormal operation occurs in the component.
12. The method of claim 11,
Wherein,
Calculating a remaining lifetime of the component to determine necessity of replacement,
And if the component needs to be replaced, generates an advertisement for the replaceable model of the component and allocates the advertisement to the augmented reality submenu.
12. The method of claim 11,
Wherein the augmented reality interface includes an augmented reality representative icon of a different shape or color according to the type of the augmented reality submenu.
11. The method of claim 10,
A user terminal including the camera unit, the communication unit, the display unit, and the control unit;
And a server including the object recognition unit and the augmented reality processing unit
Augmented reality providing system.

Images