KR102338627B1 - System and method for providing augmented reality user interface - Google Patents

Abstract

The present invention relates to a system and method for providing augmented reality. The augmented reality providing method includes: photographing an image of an object through a first camera included in the user terminal, recognizing an object included in the photographed image, and using a second camera included in the user terminal capturing an image of the user through the steps of, recognizing a user's face element included in the captured image, determining the user's access right to the object based on the recognized face element, and the and displaying an augmented reality interface for the object overlapping the captured image when the user has access to the object in the user terminal.

Description

System and method for providing augmented reality user interface

The present invention relates to a system and method for providing augmented reality. Augmented reality providing system and method

Augmented reality (AR) refers to a computer graphics technique in which virtual objects or information are synthesized in a real environment to make them look like objects existing in the original environment.

Unlike conventional virtual reality, which targets only virtual space and objects, augmented reality has a feature of providing additional information that is difficult to obtain only in the real world by synthesizing virtual objects on the basis of the real world. Because of these characteristics, augmented reality can be applied to a variety of real environments, unlike existing virtual reality, which can be applied only to fields such as games. In particular, augmented reality is spotlighted as a next-generation display technology suitable for a ubiquitous environment.

Since the research on augmented reality technology began, research to utilize augmented reality technology has been continued in various fields.

In general, augmented reality technology displays information about a device currently being photographed on a preview screen in the form of a menu. In this case, the preview screen may display status information of the device being photographed in 2D or 3D form.

However, in the case of the conventional augmented reality providing method, the same augmented reality is provided to all users regardless of the user's access rights. Therefore, augmented reality is provided even to a user who does not have access to the device, and there is a problem in that the security of the device is weakened.

In addition, in augmented reality, the target device must be photographed through the user terminal, and the user must hold the user terminal by hand. In this case, when input of a password is required to authenticate access authority of the target device, there is a problem in that it is difficult for the user to input the password while holding the user terminal with one hand.

An object of the present invention is to provide an augmented reality providing system and method for displaying an augmented reality interface of only a target device to which a current user has access right among target devices included in a captured image.

In addition, the present invention performs authorization authentication of the photographed target device through the user's face recognition without any special manipulation while the target device is being photographed, and overlaps the augmented reality interface only for the target device to which the user has access rights to display the screen. An object of the present invention is to provide an augmented reality providing system and method for displaying.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the appended claims.

The augmented reality providing method of the present invention for achieving the above object includes the steps of photographing an image of an object through a first camera included in the user terminal, recognizing an object included in the photographed image, the Taking an image of the user through a second camera included in the user terminal, recognizing a user's face element included in the captured image, based on the recognized face element, the user's Determining an access right, and when the user has an access right to the object in the user terminal, overlapping and displaying an augmented reality interface for the object on the captured image.

In addition, the step of recognizing the user's face element included in the photographed image includes extracting the user's face image included in the photographed image, and at least one facial element of the user from the extracted facial image It may include the step of extracting.

In addition, the face element may include a shape of the user's iris, face outline, or features.

In addition, the determining of the user's access right to the object based on the recognized face element may include comparing the recognized face element with a face element of a registered user input in advance with respect to the object, and the user and determining whether is a registered user.

In addition, the determining of the access right to the object based on the recognized facial element may include: data of a registered user in which data on the first and second different facial elements of the user are inputted in advance for the object and determining whether the user is a registered user based on whether or not they all match.

In addition, the step of overlapping and displaying the augmented reality interface for the object on the captured image includes the steps of requesting an augmented reality interface for the object to which the user has an access right to a server connected to the user terminal, the augmented reality The method may include overlapping and displaying the augmented reality image included in the interface on the object having the access right.

In addition, the displaying of the augmented reality interface for the object overlapping the captured image may include displaying the augmented reality interface including different information according to the user's access right level.

In addition, the step of displaying the augmented reality interface for the object overlapping the captured image may further include displaying the captured image as it is in the case of an object to which the user does not have access right.

In addition, when the user does not have access to the object, the method may further include moving to a permission registration menu for registering the user's access right to the object.

Augmented reality providing system of the present invention for achieving the above object, a first camera unit for photographing an image of an object, a second camera unit for photographing an image of a user, the image captured by the first camera unit An object recognition unit for recognizing the included object, a face recognition unit for recognizing the user's face element photographed by the second camera unit, and an augmented reality processing unit for generating an augmented reality interface based on augmented reality information on the object , based on whether the user has access right determined using the augmented reality information on the object or a communication unit for receiving the augmented reality interface, a display unit for displaying the augmented reality interface, and the recognized face element, the and a controller for displaying an augmented reality interface for the object.

In addition, the first camera unit may be disposed on one side of the augmented reality providing device, and the second camera unit may be disposed on the other side facing the one side of the augmented reality providing device.

Also, the controller may determine whether the user is a registered user by comparing the recognized face element with a face element of a registered user previously input with respect to the object.

In addition, when the user has access to the object, the controller requests an augmented reality interface for the object from a server connected to the user terminal and displays it on the display unit, and provides the user with information about the object. If there is no access right, the image of the photographed object may be displayed on the display unit as it is.

In addition, when the user does not have an access right to the object, the controller may display a permission registration menu for registering the user's access right to the object on the display unit.

Also, the face recognition unit may recognize a shape of the user's iris, face outline, or features as the face element.

In addition, a user terminal including the first and second camera units, the communication unit, the display unit, and the control unit, and a server including the object recognition unit and the augmented reality processing unit.

The augmented reality providing system and method of the present invention can prevent leakage of important information and improve the security of the system by displaying the augmented reality interface only for the target device to which the current user has access.

In addition, the augmented reality providing system and method of the present invention can omit a process in which the user must directly input a password by performing authorization authentication of the target device through the user's face recognition without special manipulation while the target device is being photographed. have. Through this, it is possible to increase the user's convenience, and since it is possible to quickly process a task within the same time, work efficiency can be improved.

In addition to the above-described effects, the specific effects of the present invention will be described together while describing specific details for carrying out the invention below.

1 is a diagram illustrating an augmented reality providing system according to an embodiment of the present invention.
2 is a block diagram illustrating an apparatus for providing augmented reality according to an embodiment of the present invention.
3 and 4 are diagrams for explaining a method of operating an apparatus for providing augmented reality according to an embodiment of the present invention.
5 is a flowchart illustrating a method for providing augmented reality according to an embodiment of the present invention.
6 is a flowchart illustrating an example of an operation of the augmented reality providing method of FIG. 5 .
7 is a flowchart illustrating another example of the operation of the augmented reality providing method of FIG. 5 .
8 to 10 are diagrams for explaining an example of an operation of an augmented reality providing method according to some embodiments of the present invention.

The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

Hereinafter, an augmented reality providing system and method according to the present invention will be described with reference to FIGS. 1 to 10 .

1 is a diagram illustrating an augmented reality providing system according to an embodiment of the present invention.

Referring to FIG. 1 , the augmented reality providing system according to an embodiment of the present invention includes a user terminal 100 , a target device 200 , and a server 300 .

The user terminal 100 may operate as an augmented reality providing device. The user terminal 100 photographs the target device 200 and receives the augmented reality interface of the target device 200 from the server 300 . The user terminal 100 displays different augmented reality interfaces according to the user's access rights.

For example, after determining the user's access right to the target device 200 to be photographed, the user terminal 100 may display the augmented reality interface for the target device 200 having the access right to the user. Through this, it is possible to improve security by preventing the provision of important information to users who do not have access rights.

Also, the user terminal 100 may photograph the user's face and determine whether access to the target device 200 is authorized through face recognition. The user terminal 100 may perform authorization authentication for the target device 200 by recognizing the user's face without special manipulation while the target device 200 is being photographed. Accordingly, the augmented reality providing system of the present invention can increase user convenience by omitting a process in which the user must separately input a password for authorization authentication.

The user terminal 100 receives augmented reality information about the target device 200 to be photographed from the server 300 , generates an augmented reality interface based on the augmented reality information, and displays it on the screen. Here, the augmented reality information includes augmented reality related data about the captured target device 200 for generating the augmented reality interface.

Augmented reality information may be modeled and rendered as a 2D or 3D image, and the rendered image is converted into an augmented reality interface and displayed in a preview form on the screen of the user terminal 100 .

Additionally, the user terminal 100 may receive the augmented reality interface for the target device 200 to be photographed from the server 300 and display it on the screen as it is.

The user terminal 100 may include a mobile-based device, a wearable device (eg, a head mounted device (HMD), a video see-through device), It may be any one of an optical see-through device, a screen-based device, and a projector-based device.

In addition, the user terminal 100 is a computer, an Ultra Mobile PC (UMPC), a workstation, a net-book, a PDA (Personal Digital Assistants), a portable computer, a web tablet (web tablet), a wireless phone ( It may be a device capable of transmitting and receiving information such as a wireless phone, a mobile phone, a smart phone, an e-book, and a portable multimedia player (PMP) in a wireless environment. However, the present invention is not limited thereto.

The target device 200 may be various electronic devices. For example, the target device 200 may be an automatic control device, an industrial device, or a switchboard. In this case, the target device 200 may include a circuit breaker, a relay, a communication link device, a watt-hour meter, a sensor unit, a human machine interface (HMI), etc. as components, and the operating state of the components included in the target device 200 . Accordingly, the overall operating state of the target device 200 may be determined.

As another example, the target device 200 may be a measurement target of an automatic meter reading system (hereinafter, AMR). In general, the remote meter reading system (AMR) refers to converting a measurement result into an electrical signal from a measurement target at a distant point, transmitting data through a communication network, and processing the data in a computer.

The server of the remote meter reading system (AMR) uses a data communication network (eg, backbone network, private network, Internet network included) provided to perform data communication based on TCP/IP (Transmission Control Protocol/Internet Protocol). interconnected with the collection device. The server performs data transmission and reception in real time with the data collection device through the data communication network, and it is possible to monitor the power state of each measurement target in real time.

In addition, the server receives the meter reading data of each measurement target in real time from the data collection device. Based on the meter reading data, the power state of each measurement target, for example, a power failure, conduction, leakage state, etc. is analyzed and provided to the manager.

Here, the meter reading data may be the sum total of the amount of power used by all measurement targets allocated to one data concentrator device, the amount of power used by each measurement target, the instantaneous power amount, etc. In addition, all data that can determine the power quality of each measurement target may be included.

The server 300 includes a database for storing various types of information on the target device 200 . For example, the server 300 may include object information on the target device 200 , status information of the target device 200 , information on components included in the target device 200 , and access rights of the target device 200 . Information and data related to augmented reality for the target device 200 may be stored.

The server 300 may operate as a data collection device or server of the remote meter reading system (AMR). That is, the server 300 may receive and store the meter reading data for the target device 200 in real time. Through this, the server 300 may monitor and remotely control the operating state of the target device 200 . However, the present invention is not limited thereto.

2 is a block diagram illustrating an apparatus for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 2 , the user terminal 100 as an augmented reality providing apparatus according to an embodiment of the present invention includes a controller 110 , a camera unit 120 , an object recognition unit 130 , and a face recognition unit 135 . , a position measuring unit 140 , an augmented reality processing unit 150 , a communication unit 160 , a sensor unit 165 , a memory unit 170 , a display unit 180 , and an interface unit 190 . However, since the components of the user terminal 100 shown in FIG. 2 are not essential, the user terminal 100 having more components or fewer components may be implemented.

The controller 110 may control the overall operation of the components included in the user terminal 100 . The controller 110 may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the controller 350 may access, store, manipulate, process, and generate data in response to the execution of the software. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media. A detailed operation of the control unit 110 will be described below.

The camera unit 120 is a device for capturing an image, and the captured image is transmitted to the controller 110 . The image captured by the camera unit 120 may be displayed on the display unit 180 in the form of a real-time preview.

The camera unit 120 may include a front camera unit 122 and a rear camera unit 124 . The front camera unit 122 is disposed on the same plane as the display unit 180 of the user terminal 100 and may be used to photograph a user's face. The rear camera unit 124 is disposed on the rear side of the user terminal 100 , and may be used to photograph the target device 200 . In this case, the front camera unit 122 may be disposed on one surface of the user terminal 100 , and the rear camera unit 124 may be disposed on the other surface facing the one surface of the user terminal 100 .

The object recognition unit 130 recognizes an object included in the image captured by the camera unit 120 . The object recognition method used in the object recognition unit 130 includes a marker-based recognition method for recognizing an object using a marker included in an image, and a markerless recognition method for recognizing a pre-registered real object instead of a physical marker.

Specifically, the marker-based recognition method is a method of recognizing an object based on a marker such as a QR code, a data matrix, or a 2D barcode. The marker-based recognition method utilizes a special type of marker that the system understands in advance, so the calculation speed and recognition rate are high.

The markerless recognition method is divided into a natural image-based method and an environment recognition-based method. The natural image-based method analyzes a number of feature points existing in a general flat image and uses the analyzed feature points as invisible markers. The natural image-based method uses a technique for extracting a feature point existing in a natural image and a technique for effectively converting the feature point into a descriptor. The environment recognition-based method is a technique for recognizing the environment itself in the form of a three-dimensional map without a special type of marker or a pre-input image. In the environment recognition-based method, a mapping technology that creates a three-dimensional environment map in real time by analyzing data from a camera or RGBD sensor, and a camera tracking technology that estimates the current camera position on the created map can be used. have.

However, the operation of the object recognition unit 130 may be performed in a separate external device (eg, the server 300 ), and only result data of the execution result may be received and used by the user terminal 100 . .

The face recognition unit 135 captures the user's face and extracts the user's face element from the captured image. For example, the face recognition unit 135 captures the user's face through the front camera. Next, the face recognition unit 135 may extract a face element including the size, shape, and arrangement of the user's iris, face contour, or feature, and a three-dimensional pattern of the face from the captured image. In this case, the face recognition unit 135 may simultaneously extract a plurality of face elements. The facial element extracted from the face recognition unit 135 may be used to recognize the user, and may be used as basic data for determining the user's access right.

The object recognition unit 130 may recognize an object included in a captured image using the marker-based recognition method and the markerless recognition method described above. In this case, the object recognition unit 130 may recognize an object included in the image based on location information at which the image was captured, a marker for the object, or markerless information of the object. However, the method of recognizing the object by the object recognition unit 130 is not limited thereto, and various newly developed methods may be applied.

The position measuring unit 140 measures the current position of the user terminal 100 and the position of the object to be photographed. Specifically, the location measurement unit 140 may measure the current location of the user terminal 100 and the location of an object photographed by the user terminal 100 using a local location tracking technology. Depending on the method used, local location tracking technology includes time of flight (TOF), spatial scan, inertial sensing, mechanical linkage, phase difference sensing, and direct field detection. (direct field sensing), and a mixture of the above techniques can be used to increase accuracy.

The location measurement unit 140 may measure a location using a location-based service using near field communication (NFC). For example, the location-based service may use Zigbee and iBeacon technology. The location-based service can be mainly used in an indoor environment where GPS cannot be used, and the relative location can be determined through the signal strength of each node. In addition, the position measurement unit 140 may increase the accuracy of position measurement by using various sensors (inertial measurement sensor, altimeter, compass) installed in the user terminal 100 .

The augmented reality processing unit 150 may generate an augmented reality interface provided to the user based on the augmented reality information. Augmented reality information is raw data used to model and render a photographed object. The augmented reality information may be stored in the database of the server 300 or stored in the memory unit 170 of the user terminal 100 and used.

The augmented reality processing unit 150 may use a modeling technique that virtually accurately expresses a real object based on augmented reality information, and a rendering technique that realistically expresses it. The modeling technology is a technology for synthesizing a 2D image input as a video by reflecting depth information. Modeling technology enables various applications that cannot be achieved in real space by performing object transformation and color change according to the user's intention. Rendering technology is a technology that allows a user to feel alive in the augmented reality interface by realistically expressing the virtual model by adding lighting effects, shadow effects, and brightness changes to the generated virtual models.

However, the operation of the augmented reality processing unit 150 may be performed in a separate external device (eg, the server 300 ), and only result data for the execution result may be received and used by the user terminal 100 . .

The communication unit 160 exchanges data with the server 300 that stores and manages the augmented reality database or the target device 200 that is a photographing target. In this case, the augmented reality database is an augmented reality object and a database storing attribute information of the augmented reality object.

The communication unit 160 may include an electronic device, a module, or a program that transmits and receives data to and from another terminal device in a wired or wireless manner. The communication protocol used by the communication unit 160 includes, for example, CDMA (Code Division Multiple Access) communication, WCDMA (Wideband Code Division Multiple Access) communication, or broadband wireless communication. However, the present invention is not limited thereto.

The sensor unit 165 is a device for detecting information necessary for augmented reality or context information additionally applied to augmented reality. The sensor unit 165 may include a temperature sensor, a humidity sensor, a position sensor, a motion sensor, and the like. Here, a GPS sensor that detects a GPS signal may be used as the location sensor, and a gyroscope sensor or an acceleration sensor may be used as the motion sensor. In addition, the sensor unit 165 may measure the current position, movement, tilt, direction, distance to the object to be photographed, ambient sound, brightness, and the like of the user terminal 100 .

The memory unit 170 stores an operating system for controlling the overall operation of the user terminal 100, an application program, and data for storage (phone number, SMS message, compressed image file, video, etc.). In addition, the memory unit 170 may store the augmented reality object corresponding to the object to be photographed, attribute information of the augmented reality object, and state information. That is, the memory unit 170 stores the object to be photographed, state information about the components included in the object, and augmented reality information.

The memory unit 170 may include a random access memory (RAM) or read only memory (ROM) and a static storage device for storing static information and instructions. The memory unit 170 may be a volatile memory unit or a non-volatile memory unit. Memory unit 170 may also be another type of computer readable medium, such as a magnetic or optical disk.

The display unit 180 displays status information (or indicators) generated during operation of the user terminal 100 , limited numbers and characters, a moving picture, a still picture, and the like. Also, the display unit 180 may display an image received through the camera unit 120 and display an augmented reality interface corresponding to a photographed object.

The interface unit 190 may receive an input from a user. The interface unit 190 may include an input device such as a button type input unit, a touch screen, a keyboard, or a mouse. For example, the touch screen may be formed on the upper surface of the display unit 180 and may receive an input through a user's touch. However, the present invention is not limited thereto.

3 and 4 are diagrams for explaining a method of operating an apparatus for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 3 , the user terminal 100 , which is an augmented reality providing apparatus according to an embodiment of the present invention, uses a rear camera unit 124 to photograph a target device 200 , and the photographed target device 200 . displays the augmented reality interface for the on the display unit 180 .

Here, the target device 200 may include first and second target devices 210 and 220 . For example, each of the first and second target devices 210 and 220 may be any one of an automatic control device, an industrial device, or a switchboard. However, the present invention is not limited thereto.

The server 300 works with the target device 200 to store various data about the target device 200 . The server 300 includes an object information DB 310 , an access right information DB 320 , and an augmented reality DB 330 . Hereinafter, the target device 200 will be described by mixing it with the term 'object', which is a photographing target.

The object information DB 310 includes various information about objects. For example, it may include information such as product information of the object (product name, product number, product type, installation time, management method, etc.), location information where the object is installed, and a circuit diagram related to the object.

The access authority information DB 320 includes access authority information of a user who is granted access to an object. The access authority information DB 320 stores the user's access authority information for each object. In this case, the access right information DB 320 may store biometric information that can identify the user, such as the face element and fingerprint of the registered user for each object, and a password and encryption key preset by the user. That is, the access authority information DB 320 stores information necessary for determining whether the user has access authority to the object. In this case, different levels of access rights for objects may be assigned according to users. That is, each user may have different access authority levels, and the level of information that can be accessed for an object may be different according to the user's access authority level.

The augmented reality DB 330 stores augmented reality information that is raw data used to model and render an object. In addition, the augmented reality DB 330 stores the augmented reality interface provided to the user based on the augmented reality information. That is, the augmented reality DB 330 may store all of the augmented reality information before modeling and rendering of the object and the augmented reality interface after modeling and rendering, and may provide necessary data according to the request of the user terminal 100 . .

The user terminal 100 recognizes the target device 200, which is an object included in the captured image, and receives augmented reality information or an augmented reality interface for the target device 200 from the server 300, and the display unit 180 displayed on In this case, the user terminal 100 determines the current user's access right to the target device 200 or receives information about the user's access right from the server 300 . Next, the user terminal 100 displays only the augmented reality interface for the target device 200 to which the user has access right on the display unit 180 .

Referring to FIG. 4 , the user terminal 100 captures an image of a user's face. At this time, the user terminal 100 uses the front camera unit 122 to photograph the user's face, and extracts the user's face elements from the captured image. For example, the user terminal 100 captures the user's face, recognizes the user's iris S11 and S12 from the captured image, and then, based on the recognized data, the user selects the target device 200 . Determine if you have access to However, the present invention is not limited thereto, and the user terminal 100 may consider the user's face contour or a three-dimensional pattern of the user's face among the user's face elements, and based on the size, shape and arrangement of each feature You can determine the user's access right.

A detailed description of the augmented reality providing method performed in the user terminal 100 will be described below.

5 is a flowchart illustrating a method for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 5 , in the augmented reality providing method according to an embodiment of the present invention, first, the user terminal 100 uses the rear camera unit 124 to display an image including an object (ie, the target device 200 ). is photographed (S110).

Next, the object recognition unit 130 recognizes an object included in the captured image (S120). The object recognition unit 130 may recognize all of the plurality of objects included in the captured image.

Next, the user terminal 100 captures an image including the user using the front camera unit 122, and recognizes the user from the captured image (S130).

Then, the user terminal 100 transmits the user recognition information to the server 300 (S140). For example, the user terminal 100 may recognize a user's face and transmit user recognition information including a face element included in the face to the server 300 .

Next, the server 300 compares the registered user information input in advance for the specific object with the user recognition information received from the user terminal 100 to determine the access right to the object (S150). In this case, the subject of determining the access right to the object is the server 300 , and the user terminal 100 may receive a result of whether or not the access authority is present.

In another embodiment of the present invention, although not clearly shown in the drawings, the user terminal 100 receives the pre-entered registered user information about the object from the server 300, and directly determines whether access to the object is authorized. can

Next, the user terminal 100 displays the augmented reality interface for the object on the display unit 180 when the current user has the access right to the object in consideration of the access right to the photographed object (S160, S170) . In this case, the user terminal 100 may receive and display the augmented reality interface for the object from the server 300 .

On the other hand, if the current user does not have an access right to the photographed object, the user terminal 100 moves to a permission registration menu for registering the user's access right to the object (S165).

Subsequently, if the user still does not have access to the corresponding object even after that, the user terminal 100 may display the image captured for the corresponding object as it is.

Additionally, in an embodiment of the present invention, the user terminal 100 may convert the augmented reality related data received from the server 300 into an augmented reality interface and output the converted data to the display unit 180 . The augmented reality processing unit 150 of the user terminal 100 may additionally apply effects such as object transformation or color change by reflecting a preset value set by the user.

Additionally, in another embodiment of the present invention, the user terminal 100 receives the augmented reality interface stored in advance in the server 300 for the target device 200, which is an object included in the captured image, and directly displays the augmented reality interface on the display unit 180. can be displayed

For example, the server 300 may perform the operations of the object recognition unit 130 and the augmented reality processing unit 150 of the user terminal 100 described above. Subsequently, the user terminal 100 may receive and use result data of the operation performed from the server 300 . Specifically, the user terminal 100 transmits the image captured by the rear camera unit 124 to the server 300, and the server 300 recognizes an object included in the received image, and augments the recognized object. A real interface may be generated and provided to the user terminal 100 .

In this case, the user terminal 100 can use the data (ie, the augmented reality interface) received from the server 300 as it is, so that the amount of computation of the control unit 110 is reduced, so that the operation speed can be increased.

In addition, the user terminal 100 may output the augmented reality interface changed to suit the user's request by reflecting the user's setting value. That is, the user terminal 100 may not only use the augmented reality interface provided by the server 300 as it is, but also implement various expression effects desired by the user on the augmented reality interface.

Accordingly, the augmented reality providing method according to another embodiment of the present invention may change and output the augmented reality interface to suit the needs of the user.

6 is a flowchart illustrating an example of an operation of the augmented reality providing method of FIG. 5 . In FIG. 6, content overlapping with the content described with reference to FIG. 5 will be omitted, and differences will be mainly described.

Referring to FIG. 6 , following step S120 described above, the user terminal 100 takes a face image of the user using the front camera unit 122 ( S230 ).

Next, the user terminal 100 extracts a face element included in the photographed face image ( S240 ). Specifically, the face recognition unit 135 of the user terminal 100 recognizes the user's face in the captured image. Then, the user terminal 100 recognizes a face element including the size, shape and arrangement of each of the user's iris, facial contour, or feature, and a three-dimensional pattern of the face. The recognized user's face element is used to authenticate the user and determine the user's access right.

Next, the user terminal 100 performs user authentication based on the recognized face element of the current user (S250). In this case, the subject of user authentication may be the user terminal 100 or the server 300 . User authentication is performed based on whether the face element of the registered user input in advance for a specific object and the recognized face element of the current user match. If the registered user for the object matches the face element of the current user, the user terminal 100 or the server 300 determines that the current user has access to the object.

Next, the user terminal 100 checks whether the recognized user has access to each of the recognized objects (S260).

Then, the user terminal 100 overlaps and displays only the augmented reality interface for the object whose access right is confirmed on the display unit 180 (S270). In this case, the user terminal 100 may overlap and display different augmented reality interfaces on the object according to the user's access right level. For example, an augmented reality interface including only limited information about the object may be displayed to a user with a low access permission level, and an augmented reality interface including all information about the object may be displayed to a user having a high access permission level. have. However, this is only an example, and the present invention is not limited thereto.

That is, the user terminal 100 may determine whether access to the photographed object is right based on the face element of the current user, and display only the augmented reality interface for the object to which the current user has access right on the screen.

7 is a flowchart illustrating another example of the operation of the augmented reality providing method of FIG. 5 . In FIG. 7, content overlapping with the content described with reference to FIG. 5 will be omitted, and differences will be mainly described.

Referring to FIG. 7 , following step S120 described above, the user terminal 100 captures a face image of the user using the front camera unit 122 and extracts a plurality of face elements included in the captured face image. (S330).

Specifically, the face recognition unit 135 of the user terminal 100 recognizes the user's face image from the captured image. Next, the user terminal 100 extracts a plurality of face elements from the face image. In this case, the face element may include the user's iris, facial contour, or the structure, arrangement, and shape of the features.

Next, the user terminal 100 or the server 300 performs user authentication using a plurality of face elements for the current user ( S340 ). Specifically, the user terminal 100 or the server 300 determines whether the data for the first and second face elements different from the current user match all the data of the registered user input in advance for the corresponding object, based on whether It may be determined whether the current user is a registered user. For example, the first facial element may be the iris, and the second facial element may be facial contour information. However, the present invention is not limited thereto.

Next, the user terminal 100 confirms the access right of the current user for each of the plurality of photographed objects ( S350 ).

Next, the user terminal 100 displays only the augmented reality interface for the object for which the user's access right is confirmed among the plurality of photographed objects on the display unit 180 (S360).

8 to 10 are diagrams for explaining an example of an operation of an augmented reality providing method according to some embodiments of the present invention.

Referring to FIG. 8 , in the augmented reality providing method according to some embodiments of the present invention, the user terminal 100 captures a face image of the current user through the front camera unit 122 .

Next, the user terminal 100 extracts a face element from the photographed face image of the user. For example, the face recognition unit 135 of the user terminal 100 first recognizes the user's face image S2 from the captured image. Next, the face recognition unit 135 recognizes the eye portion S1 from the face image S2 and extracts data for both irises S11 and S12 .

The user terminal 100 compares the data on both iris S11 and S12 of the user with the data of the registered user for the pre-entered object. The registered user's data may be stored in the server 300 , and may be transmitted to the user terminal 100 by a call of the user terminal 100 . Through this, the user terminal 100 may determine whether the user has an access right to the photographed target object.

At this time, although the user terminal 100 has been described as the subject for determining the user's access right, the present invention is not limited thereto. For example, the user terminal 100 transmits the extracted data on both iris S11 and S12 of the user to the server 300, and the server 300 based on the registered user data for the pre-entered object. to determine the user's access right. Subsequently, the server 300 may transmit the user's access right to the determined object to the user terminal 100 .

Here, an embodiment in which only the iris is recognized as the user's face element has been described, but the present invention is not limited thereto, and the user's Access rights can be determined.

In this case, the user may perform the user's authority authentication while stably holding the user terminal 100 with both hands. Accordingly, the augmented reality providing system and method of the present invention may omit a process of separately inputting a password by performing authorization authentication for an object through face recognition without special manipulation while photographing the object. Through this, user's convenience can be improved, and work efficiency can be improved because quick work processing is possible within the same time.

Referring to FIG. 9 , in the augmented reality providing method according to some embodiments of the present invention, the user terminal 100 photographs a target device 200 , and the target device 200 includes first and second objects 210 , 220) may be included. The user terminal 100 determines whether the current user has access rights to the first and second objects 210 and 220 .

If the current user of the user terminal 100 has an access right to the first object 210 , the user terminal 100 is augmented reality for the first object 210 on the first object 210 . interface can be displayed. The user terminal 100 may display the augmented reality image included in the augmented reality interface to overlap the first object 210 . In this case, the augmented reality menu M31 indicating that the user has the access right may be displayed on the first object 210 .

On the other hand, when the current user of the user terminal 100 does not have access to the second object 220 , the user terminal 100 may display the second object 220 as it is a captured image. In this case, the augmented reality menu M21 indicating that the user does not have an access right may be displayed on the second object 220 .

In addition, the user terminal 100 may display the augmented reality menu M22 on the screen in which the user can apply for permission registration for the second object 220 . When the user selects the augmented reality menu M22 for the permission registration application, the user terminal 100 screens to a homepage or interface including a permission registration menu for registering user rights for the second object 220 can be switched

Referring to FIG. 10 , a user may select a first object 210 to which an augmented reality interface is provided. In this case, the user terminal 100 may display the augmented reality menu M32 indicating detailed information on the first object 210 .

For example, when the user selects the first object 210 , the augmented reality processing unit 150 provides the first object 210 with operational state and state information (eg, amount of operational power, amount of change in power per hour, and average amount of power) of the first object 210 . , electricity amount change trend, etc.) can be generated and displayed in an augmented reality interface. The user terminal 100 displays detailed information about the object in the augmented reality interface only when the user has access to the object.

In this case, the user terminal 100 may overlap the first object 210 and display different augmented reality menus M32 according to the user's access authority level. For example, an augmented reality menu (M32) including only limited information about the object is displayed to a user with a low access permission level, and an augmented reality menu (M32) including all information about the corresponding object to a user with a high access permission level ( M32) can be displayed. However, this is only an example, and the present invention is not limited thereto.

On the other hand, when the user selects the second object 220 , the user terminal 100 does not display any information about the second object 220 . That is, when the user does not have access to the object, the user terminal 100 does not display information on the object, but displays the image captured by the camera for the object as it is.

Through this, since the augmented reality providing system and method of the present invention provides the augmented reality interface of the device only to a user with access rights, it is possible to prevent leakage of important information and improve security.

Although embodiments of the present invention have been described with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: user terminal
200: target device
300: server

Claims (16)

In the augmented reality providing method performed in a user terminal or server,
photographing an image of an object through a first camera included in the user terminal;
recognizing an object included in the captured image;
photographing an image of a user photographing the object through a second camera included in the user terminal;
recognizing a user's face element included in the captured image;
determining an access right of the user to the object based on the recognized face element; and
In the user terminal, when the user has access to the object, displaying an augmented reality interface for the object overlapping the captured image,
The first camera is disposed on one surface of the user terminal,
The second camera is disposed on the other surface facing the one surface of the user terminal
How to provide augmented reality.
According to claim 1,
The step of recognizing the user's face element included in the captured image,
extracting the user's face image included in the captured image;
and extracting at least one facial element of the user from the extracted facial image.
3. The method of claim 2,
The facial element is an augmented reality providing method including the shape of the user's iris, facial contour, or features.
According to claim 1,
The step of determining the user's access right to the object based on the recognized face element comprises:
and determining whether the user is a registered user by comparing the recognized face element with a face element of a registered user previously input with respect to the object.
5. The method of claim 4,
The step of determining the access right to the object based on the recognized face element,
Determining whether the user is a registered user based on whether the data on the first and second different facial elements of the user match all of the registered user data input in advance for the object How to provide augmented reality.
According to claim 1,
The step of overlapping and displaying the augmented reality interface for the object on the captured image,
requesting an augmented reality interface for an object to which the user has access rights to a server connected to the user terminal;
Augmented reality providing method comprising the step of overlapping and displaying the augmented reality image included in the augmented reality interface on the object having the access right.
According to claim 1,
The step of overlapping and displaying the augmented reality interface for the object on the captured image,
Augmented reality providing method comprising displaying an augmented reality interface including different information according to a user's access right level.
According to claim 1,
The step of overlapping and displaying the augmented reality interface for the object on the captured image, Augmented reality providing method further comprising the step of displaying the captured image as it is in the case of an object that the user does not have access to.
According to claim 1,
If the user does not have access to the object, the augmented reality providing method further comprising the step of moving to a permission registration menu for registering the user's access right to the object.
a first camera unit for capturing an image of an object;
a second camera unit that takes an image of a user who shoots the object;
an object recognition unit for recognizing the object included in the image captured by the first camera unit;
a face recognition unit recognizing the user's face element photographed by the second camera unit;
an augmented reality processing unit generating an augmented reality interface based on the augmented reality information on the object;
a communication unit configured to receive augmented reality information on the object or the augmented reality interface;
a display unit displaying the augmented reality interface; and
a control unit for displaying an augmented reality interface for the object based on whether the user has access right determined using the recognized face element;
The first camera unit is disposed on one surface of the augmented reality providing device,
The second camera unit is disposed on the other surface facing the one surface of the augmented reality providing device
Augmented reality providing system.
delete 11. The method of claim 10,
The controller is configured to compare the recognized face element with a face element of a registered user previously input with respect to the object to determine whether the user is a registered user.
11. The method of claim 10,
The control unit is
When the user has access to the object, an augmented reality interface for the object is requested from a server connected to the user terminal and displayed on the display unit,
When the user does not have access to the object, the augmented reality providing system for displaying the image of the photographed object as it is on the display unit.
11. The method of claim 10,
The control unit, when the user does not have the access right to the object, the augmented reality providing system for displaying a permission registration menu for registering the user's access right to the object on the display unit.
11. The method of claim 10,
The face recognition unit, an augmented reality providing system for recognizing the shape of the user's iris, face outline, or features as the face element.
11. The method of claim 10,
a user terminal including the first and second camera units, the communication unit, the display unit, and the control unit;
Containing a server including the object recognition unit and the augmented reality processing unit,
Augmented reality providing system.

Images