KR101945294B1 - Augmented reality system that can identify similar objects - Google Patents

Abstract

The present invention relates to an augmented reality system capable of identifying a similar object. The augmented reality system includes: augmented reality glasses provided therein with a video camera for acquiring actual image information by capturing an image of a user′s surroundings, in which the augmented reality glasses display 3D virtual image on a transparent display such that the 3D virtual image corresponding to current position information and the actual image information is displayed within a visual range of the user; and a server for providing the augmented reality glasses with the 3D virtual image corresponding to the current position information and the actual image information transmitted from the augmented reality glasses in real time. The server identifies an actual object of the real image information and provides a virtual object of the 3D virtual image assigned to the identified real object to the augmented reality glasses. The real objects having a predetermined image similarity degree value or more among the plurality of actual objects existing in the actual image information may select an additional recognition area by subtracting each actual image information, and a unique identifier is assigned to the actual object based on the image difference of the additional recognition area.

Description

[0001] The present invention relates to an Augmented Reality System

The present invention relates to an augmented reality system, and more particularly, to an augmented reality system capable of identifying a similar object to prevent duplicate recognition.

Recently, contents are being actively researched using augmented reality technique in which various information is superimposed on a real space when photographing using a camera module.

Augmented Reality (AR) is a technology belonging to one of the fields of Virtual Reality (VR). It combines a virtual environment with a real environment that a user feels in a sense, and feels as if a virtual environment exists in the original real environment Is a computer technique.

This augmented reality is advantageous in that it can supplement and supplement additional information that is difficult to obtain only in the real world by synthesizing virtual objects on the basis of the real world, unlike the existing virtual reality which only targets virtual space and objects.

That is, a realization realized by matching a virtual environment created by computer graphics to a real image viewed by a user, for example, a three-dimensional virtual environment, can be called an augmented reality. Here, the 3D virtual environment provides necessary information from the real image that the user views, and the 3D virtual image matches with the real image to increase the user's immersion.

Such augmented reality can provide better realism by making the distinction between the real environment and the virtual environment be blurred by providing the real image in the three-dimensional virtual environment as well as the simple virtual reality technology.

Due to the advantages of such an augmented reality system, active research and development are currently under way for technologies that have been combined with augmented reality around the world. For example, augmented reality technology is being put into practical use in fields such as broadcast, advertisement, exhibition, game, theme park, military, education and promotion.

Recently, the handheld augmented reality system has become possible by improving the computing ability of mobile devices such as mobile phones, personal digital assistants (PDAs), and ultra mobile personal computers (UMPCs) and wireless network devices.

As these systems become available, many augmented reality applications using mobile devices have been developed. In addition, the spread of mobile devices is becoming very common and an environment for accessing augmented reality applications is being created.

In addition, user demands for various supplementary services using an augmented reality of a terminal are increasing, and attempts to apply various augmented reality contents to a terminal user are increasing.

Korean Patent Laid-Open Publication No. 10-2016-0092292 relates to a system and method for providing an augmented reality service for an advertisement, and it is an object of the present invention to apply an augmented reality to an advertisement, And the system is able to communicate effectively by being intrigued and intriguing.

Meanwhile, various wearable device products that can be worn on the body of the user are being released. In particular, a system has been developed in which mobile technology and transparent display technology are applied to glasses to allow users to check various information through glasses.

1 is a diagram illustrating a virtual object displayed in a conventional augmented reality system.

Referring to FIG. 1, the conventional augmented reality system displays too many virtual objects in a transparent display of eyeglasses, thereby hindering the user's eyesight. For this reason, it is difficult for the user to recognize necessary information.

In addition, if there are a plurality of actual objects having similar appearance, duplicate recognition or unassigned virtual objects may be displayed to cause confusion, so that a technology capable of identifying duplicate objects and preventing duplicate recognition is required.

KR 10-2016-0092292 A

An object of the present invention is to provide an augmented reality system capable of identifying a similar object by assigning a unique identifier to an actual object based on an image difference in an additional recognition area, .

Also, there is provided an augmented reality system capable of identifying each real object by considering both the unique identifier given to each actual object and the current position information of each real object based on the image difference in the additional recognition area.

According to an embodiment of the present invention, there is provided an image display apparatus for displaying a three-dimensional virtual image on a transparent display, the apparatus including a video camera for capturing an image of a user and acquiring real image information, An augmented reality glasses for displaying the three-dimensional virtual image corresponding to the actual image information within a visual range of the user, and a display unit for displaying the three-dimensional virtual image corresponding to the current position information and the actual image information transmitted from the augmented reality glasses, The server includes a server for providing the augmented reality glasses in real time, and the server identifies actual objects of the actual image information and provides virtual objects of the three-dimensional virtual images assigned to the identified actual objects to the augmented reality glasses Wherein a predetermined image similarity degree value or more among a plurality of actual objects existing in the actual image information And a unique identifier is assigned to each of the real objects based on the image difference of the additional recognition area.

The server included in the present invention identifies each real object by considering both the unique identifier given to each actual object and the current position information of each real object based on the image difference in the additional recognition area. do.

The server included in the present invention is characterized in that a candidate position of an additional recognition area is determined based on current position information of each actual object.

In addition, the augmented reality glasses included in the present invention are characterized by providing satellite position information to the server as the current position information.

Also, the augmented reality glasses provide satellite position information to the server as the current position information, and further grasp the signal strength of at least one of the searched Wi-Fi repeaters and transmit the signal strength to the server. do.

According to another aspect of the present invention, there is provided an augmented reality system including a plurality of sensing units each including a Wi-Fi communication module and disposed at regular intervals in a room, and the plurality of sensing units are connected to a Wi- WIFI HOTSPOT) signal is transmitted to the server.

The augmented reality system according to the embodiment of the present invention can identify the similar object by selecting an additional recognition area and assigning a unique identifier to the actual object based on the image difference of the additional recognition area.

Also, the augmented reality system can identify each real object by considering both the unique identifier assigned to each real object and the current position information of each real object based on the image difference of the additional recognition area.

1 is a diagram showing a virtual object displayed in a conventional augmented reality system;
2 is a block diagram of an augmented reality system 1 according to an embodiment of the present invention.
3 is a flowchart showing a learning process for identifying similar objects in the augmented reality system 1. Fig.
3A is a diagram showing a process of selecting an additional recognition area for identifying a similar object in the AR system 1
4 is a flowchart showing a process of identifying a similar object in the augmented reality system 1
5 is a first exemplary diagram showing a state for identifying similar objects in the augmented reality system 1. Fig.
6 is a second exemplary diagram showing a state for identifying a similar object in the augmented reality system 1. Fig.
7 is a diagram showing another operation principle of the augmented reality system 1. Fig.
8 and 8A are diagrams showing still another operation principle of the augmented reality system 1
9 is a block diagram of the augmented reality glasses 100 of the augmented reality system 1. Fig.
9A is an illustration of an augmented reality glasses 100
10 is a state diagram in which the safe mode of the augmented reality system 1 is operated
11 is a diagram showing a state in which the overhead view mode of the augmented reality system 1 is operated

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, in order to facilitate a person skilled in the art to easily carry out the technical idea of the present invention.

FIG. 2 is a block diagram of an augmented reality system 1 according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating an operation principle of an augmented reality system 1. As shown in FIG.

The augmented reality system 1 according to the present embodiment includes only a brief configuration for clearly explaining the technical idea to be proposed.

2 and 3, the augmented reality system 1 includes augmented reality glasses 100, a server 200, and a plurality of sensing units 300. As shown in Fig. Here, the plurality of sensing units 300 may be selectively provided in the augmented reality system 1 according to the embodiment.

The augmented reality glasses 100 have a built-in image camera for capturing an image of a user around the user and acquiring actual image information. In displaying the 3D virtual image on the transparent display, the augmented reality glasses 100 include three- The image is displayed within the user's field of view.

The server 200 provides the three-dimensional virtual image corresponding to the current position information and the actual image information transmitted from the augmented reality glasses 100 to the augmented reality glasses 100 in real time.

The augmented reality glasses 100 are basically configured to provide satellite position information to the server 200 as current position information. When a communication module is included in the augmented reality glasses 100, not only the satellite position information but also the location of the nearby Wi-Fi repeater, the location of the base station, and the like can be additionally provided to the server 200 as current location information have.

The augmented reality glasses 100 may further determine the signal intensity of at least one of the searched Wi-Fi repeaters and transmit the satellite position information to the server 200, . That is, since the absolute position of the Wi-Fi repeater located indoors is stored in advance in the server 200, the augmented reality glasses 100 can additionally provide the unique number and signal strength of the Wi-Fi repeater searched for The server 200 can grasp the relative movement path of the augmented reality glasses 100. [

That is, the relative distance between the augmented reality glasses 100 and the Wi-Fi repeater can be confirmed by the signal strength, and the moving direction can be calculated based on the change in the signal intensity with the neighboring Wi-Fi repeater . An additional method for acquiring the current position information in the room will be described later.

Therefore, the server 200 grasps the virtual objects allocated to the respective real objects through the current position information of the user wearing the augmented reality glasses 100 and the actual image information photographed by the augmented reality glasses 100 of the augmented reality glasses 100 And transmits the information to the augmented reality glasses 100 in real time.

Meanwhile, the augmented reality system 1 according to the embodiment of the present invention can identify the similar object by selecting an additional recognition area and assigning a unique identifier to the actual object based on the image difference of the additional recognition area.

In addition, the augmented reality system 1 can identify each real object by considering both the unique identifier given to each real object and the current position information of each real object based on the image difference in the additional recognition area.

Therefore, even if real objects having high similarity are arranged, it is possible to identify them and display the virtual objects allocated to the respective real objects, thereby conveying information without confusion to the user.

The visual cognition description of an object can be divided into four steps. - DCRI (Detection, Classification, Recognition, and Identification) -

First, detection is a step that can detect whether or not an object exists.

Next, Classification is a step of knowing what classification an object is. - For example, it can be classified as human or animal -

Next, Recognition is a step to know the approximate characteristics of the object. - For example, you can find brief information about clothes that a person wears -

Finally, Identification is a step that allows to know the detailed characteristics of an object. For example, you can identify a person's face and know the number of the license plate.

The augmented reality system 1 of the present invention operates so that an identification step can be implemented to distinguish detailed characteristics of similar real objects.

For example, the augmented reality system 1 recognizes a character attached to a facility (real object) having a similar shape and assigns a unique identification number, or identifies a difference portion, and then selects the additional recognition region The 2D / 3D feature information difference of the additional recognition area, and the current position information measured by the satellite position information and the Wi-Fi signal, to distinguish real objects having high similarity values.

FIG. 3 is a flowchart illustrating a learning process for identifying similar objects in the AR system 1. FIG. 3A illustrates a process of selecting an additional recognition area for identifying similar objects in the AR system 1. FIG. .

3 and 3A, the server 200 operates to identify a real object of real image information and provide a virtual object of a three-dimensional virtual image assigned to the identified real object to the augmented reality glasses 100 .

That is, the server 200 selects the additional recognition area by subtracting the actual image information from the actual objects having the predetermined image similarity value d or more among the plurality of actual objects existing in the actual image information, And assigns a unique identifier to the actual object based on the image difference of the object.

For example, when characters or numbers different from each other are displayed in the additional recognition area, the server 200 assigns a unique identifier to each of the actual objects based on the difference of the additional recognition areas, stores the database in a database, And can transmit the assigned virtual object to the augmented reality glasses 100. [

That is, when a plurality of actual objects have a predetermined image similarity value (d) or more, the image is abstracted and the image is differentiated to set an additional recognition area (additional learning area) And each unique identifier is assigned.

FIG. 4 is a flowchart illustrating a process of identifying similar objects in the AR system 1, and FIG. 5 is a first exemplary view illustrating a state in which similar objects are identified in the AR system 1. FIG.

4 and 5, when actual image information is transmitted from the augmented reality glasses 100, the server 200 distinguishes actual objects of actual image information, and if there is no similar image If there is no actual object), virtual objects corresponding to the identified image are respectively allocated.

At this time, the server 200 compares the information of the additional recognition area with the similar image (if a similar object exists), identifies the unique identifier, and then assigns the virtual object corresponding to the unique identifier.

5, when similar facilities (actual objects) are arranged at neighboring positions, the server 200 may acquire a plurality of actual objects from the actual image information transmitted from the augmented reality glasses 100 Then, the information of the additional recognition area of each real object is compared to identify the unique identifier, and the virtual object corresponding to the unique identifier is allocated.

On the other hand, when different identification markers are printed in the additional recognition area of each facility, the shape of the identification markers can be configured as follows.

The identification marker may be configured to include a first identification marker region, a second identification marker region, a third identification marker region, and a fourth identification marker region.

That is, the identification marker recognizes the first identification marker region, the second identification marker region, the third identification marker region, and the fourth identification marker region as one identifier. That is, the augmented reality glasses 100 basically capture all of the first to fourth identification markers and transmit them to the server 200, and the server 200 recognizes the recognized identification markers as one unique identifier.

At this time, the first identification marker region is configured to reflect the wavelength of the visible light region. That is, the first identification marker area is printed with a common paint and printed so that a human can visually distinguish it.

In addition, the second identification marker region reflects the first infrared wavelength, printed with a paint reflecting the first infrared wavelength, and can not be visually distinguished by a person.

Further, the third identification marker region reflects a second infrared wavelength of a wavelength longer than the first infrared wavelength, and is printed with a paint reflecting the second infrared wavelength, and can not be visually distinguished by a person.

Further, the fourth identification marker region simultaneously reflects the first infrared wavelength and the second infrared wavelength, which is printed with a paint reflecting the first and second infrared wavelengths, and can not be visually distinguished by a person.

At this time, the camera of the augmented reality glasses 100 for photographing the identification marker is configured to be able to photograph and recognize the infrared wavelength region by attaching a spectral filter for controlling the infrared transmission wavelength.

Therefore, only the first identification marker region of the identification markers printed on the facility can be visually confirmed by the person, and the second identification marker region, the third identification marker region and the fourth identification marker region can not be visually confirmed by the person, And can be photographed through the camera of the glasses 100.

Relative printing positions (left, right, top and bottom) of the first identification marker region, the second identification marker region, the third identification marker region and the fourth identification marker region can also be used as one delimiter. In the identification marker area, various characters such as numbers, symbols, and symbols may be printed. In addition, the identification marker may be printed in the form of a QR code or a bar code.

FIG. 6 is a second exemplary view illustrating a state in which similar objects are identified in the AR system 1. FIG.

Referring to FIG. 6, the server 200 can identify each real object by considering both the unique identifier given to each real object and the current position information of each real object based on the image difference in the additional recognition area . That is, the actual object can be identified by further considering the current position information of the user (the augmented reality glasses 100).

For example, when a plurality of actual objects maintain a predetermined distance, the user can identify the object using the current location information even if the actual object has a high similarity value. Herein, it is assumed that the current position information includes the absolute position information of the user, relative position information, the moving direction, the acceleration, and the direction of the line of sight.

At this time, the server 200 may select an additional recognition area to further identify actual objects not identified by the current location information, and recognize the difference to identify differences between actual objects.

Also, the server 200 can determine the candidate position of the additional recognition area based on the current position information of each actual object.

That is, referring to FIG. 6, when the user gazes at the actual object located at the first position P1 and located at the front,

The server 200 determines the distance between the real object and the user based on the actual image information, and then detects the coordinates (x1, y1, z1) on the space of the actual object.

Since the plurality of additional recognition areas are previously set in the actual object located at the coordinates (x1, y1, z1) on the space, the server 200 stores the coordinates (x1, y1, z1) The candidate position of the additional recognition area can be determined based on the current position information.

The server 200 grasps in advance which real object exists in the coordinates (x1, y1, z1) on the space and which part of the actual object is designated as the additional recognition area, It is possible to identify an object by identifying only a candidate position. Such a scheme has an advantage that the amount of calculation for identifying an additional recognition area can be reduced.

For reference, assuming indoor space and assuming that all directions of illumination are constant, the position and size of shadows due to the illumination may be different from each other even for similar real objects. Accordingly, the server 200 may identify each actual object using the difference between the shadow position and the size of each actual object as additional information based on the current position of the user.

On the other hand, the positions of the virtual objects of the three-dimensional virtual images allocated to the actual objects of the actual image information are displayed and displayed on the augmented reality glasses 100 so as to maintain a predetermined distance from the actual objects.

In addition, the positions between the virtual objects are automatically adjusted so as to maintain a predetermined separation distance, and displayed on the augmented reality glasses 100.

Therefore, since the position is automatically adjusted so that the objects are not overlapped with each other considering the mutual positional relationship between the actual object and the virtual object, the user can conveniently check the information of the desired virtual object. That is, it takes a long time to recognize the virtual objects with concentration, and the advertisement effect can be increased.

7 is a diagram showing another operation principle of the augmented reality system 1. Fig.

The operation principle of the augmented reality system 1 will be described in more detail with reference to FIG.

The distance D2 between the first virtual object (Virtual Object 1) and the actual object (Physical Object)

The distance R1 between the center point of the first virtual object 1 and the outermost area of the first virtual object 1 and the distance R1 between the center point of the actual object and the center of the physical object Is set to be longer than the sum of the distance R3 between the outer areas.

The distance Dl between the second virtual object 2 and the physical object is a distance between the center point of the second virtual object 2 and the center of the second virtual object 2 The distance R2 between the outer regions and the distance R3 between the center point of the physical object and the outermost region of the physical object.

The distance D1 between the first virtual object 1 and the second virtual object 2 is calculated by dividing the distance between the center point of the first virtual object 1 and the first virtual object 1 And a distance R2 between the center point of the second virtual object 2 and the outermost region of the second virtual object 2 (Virtual Object 2), which is longer than the sum of the distance R1 between the outermost region of the second virtual object .

The first virtual object (Virtual Object 1) and the second virtual object (Virtual Object 2) move in the horizontal direction and the vertical direction and are automatically overlapped with other objects to automatically adjust the position in the three-dimensional space so as not to deviate from the user's view. , the position is adjusted with respect to the y and z axes.

On the other hand, between the physical object and the first virtual object 1, and between the physical object and the second virtual object 2, the virtual lines L1 and L2 are dynamic As shown in FIG.

When a large number of virtual objects exist on the screen, the virtual lines L1 and L2 are displayed to indicate the association with the physical object, and the thickness, transparency, and color of the virtual lines L1 and L2 are It can be changed automatically according to the line of sight.

For example, when the user gazes at the first virtual object (Virtual Object 1) for a predetermined time or more, the augmented reality glasses 100 detects whether or not the first virtual object 1 is gazing, The thickness of the virtual line L1 between the first virtual object 1 and the first virtual object 1 is changed to be thicker than the other virtual line L2 and the transparency is also lowered, It may also operate to automatically change to color.

At this time, assuming that the first virtual object 1 and the second virtual object 2 are all allocated to the physical object, the distance of the plurality of virtual objects allocated to the physical object is As described above, virtual objects having more detailed information are held closer to a physical object while maintaining a predetermined distance D2 and D3, respectively.

For example, if the information of the first virtual object (Virtual Object 1) is more detailed information and the information of the second virtual object (Virtual Object 2) is relatively conceptual information,

The distance D3 between the second virtual object 2 and the physical object is longer than the distance D2 between the first virtual object 1 and the physical object It is automatically set up so that the user can quickly recognize detailed information.

Also, when a plurality of virtual objects are allocated to a physical object, the distance between the virtual objects is arranged closer to the virtual object, and the distance between the virtual objects is arranged more automatically have.

8 and 8A are diagrams showing another operation principle of the augmented reality system 1. Fig.

8 and 8A, the amount of information of a virtual object of a three-dimensional virtual image, which is allocated to and displayed on actual objects of actual image information, is automatically dynamically adjusted according to the distance between the actual object and the user, 100 < / RTI >

Therefore, when the user approaches the object, the virtual object displays more detailed information, so that the user can conveniently check the information of the desired virtual object. That is, the time for the user to concentrate on the virtual object is prolonged, and the information transfer effect can be increased.

In general, the user tends to look closer at the information about the object of interest, so that when the user is far away from the object of interest, the information is abstractly displayed, and when the user is close to the object of interest, .

A virtual object assigned to an actual object starts to be displayed from a time when a distance between a user and a real object or a virtual object reaches within a predetermined separation distance D1 and the closer the distance between the user and the actual or virtual object becomes A virtual object of more detailed information is displayed.

That is, information of a virtual object allocated to one real object is layered. As shown in FIG. 4A,

The most abstract virtual object A is displayed when the user enters a predetermined separation distance D1 and becomes a virtual object having more specific information when the user approaches (D2) the actual or virtual object more A1, A2) are displayed. Also, when the user approaches (D3) closest to the actual object or the virtual object, the virtual objects (A1-1, A1-2, A2-1, A2-2) having the most specific information are displayed.

For example, assuming that a vending machine is placed as a real object in front of the user,

When the user enters the predetermined separation distance D1, the virtual object allocated to the vending machine is displayed. Here, the virtual object is assumed to be the icon of the vending machine.

Next, when the user approaches the vending machine more (D2), the icons of the vending products sold in the vending machine can be displayed as virtual objects of more detailed information.

Finally, when the user closest approaches the vending machine (D3), the calories, ingredients, etc. of each beverage product can be displayed as a virtual object of more detailed information.

As another example, assuming that there is an automobile store as a real object in front of the user,

When the user enters the predetermined separation distance D1, the virtual object allocated to the car dealership is displayed. Here, the virtual object is assumed to be a car brand icon to be sold.

Next, when the user accesses the car dealer more (D2), the various kinds of car icons being sold can be displayed as virtual objects of more detailed information. At this time, a virtual object can be displayed in the vicinity of the currently displayed car, that is, an actual object if there is an actual object, and a virtual object can be displayed in the surrounding area even when the car is not displayed .

Finally, when the user closest approach (D3) to the car dealership, the specification of the vehicle being sold, the price, and the expected date of shipment can be displayed as virtual objects of more detailed information.

Meanwhile, the augmented reality glasses 100 may calculate a movement distance corresponding to a rate of change of vibration when a vibration occurs in a state where a user looks at a virtual object or an actual object to be checked, and then, The amount of information of the virtual object can be reset and displayed.

That is, it is assumed that the user has moved by virtual vibration without directly moving, or a weight is given to the movement distance by vibration, and the information amount of the virtual object is reset according to the virtual movement distance and displayed.

That is, the large change rate of the vibration means that the user is running or moving fast, and the small change rate of the vibration corresponds to the user moving slowly, so that the travel distance can be calculated based on this. Therefore, the user may be able to reflect the virtual movement distance by imparting vibration while shaking the head up and down without actually moving.

When the user gazes at a direction to be checked, for example, when the user turns his / her head to the right and continues to vibrate while looking at the right side, the augmented reality glasses 100 are moved in the direction of movement And the movement distance.

That is, the augmented reality glasses 100 detect the head rotation of the user through the built-in sensor, calculate the virtual current position after detecting the vibration, and observe the walking distance through the rate of change of vibration .

The augmented reality glasses 100 may be configured to sense the gaze direction based on the rotation direction of the head when sensing the gaze direction of the user and to sense the gaze direction by sensing the moving direction of the pupil .

Further, it is possible to simultaneously detect the rotational direction of the head and the moving direction of the pupil, and calculate the gaze direction more precisely by differentiating the specific gravity first from the two detection results. That is, it is possible to provide a specific gravity of 50% to 100% for detecting the rotation angle due to rotation of the head, and to calculate the gaze direction by giving a specific gravity of 0% to 60% to the moving direction of the pupil.

The user can also set the moving distance extension setting mode in the augmented reality glasses 100 so that the distance weight is 2 to 100 times the calculated virtual moving distance.

In calculating the moving distance corresponding to the rate of change of the vibration, the augmented reality glasses 100 except for the upper value 10% and the lower value 20% of the magnitude of the vibration are excluded so as to exclude the noise value, It is possible to calculate the moving distance.

As a result, the user can identify a virtual object that has the same amount of information as a real object or a virtual object, even if the user does not actually access the actual object or the virtual object or approaches the virtual object.

9 is a configuration diagram of the augmented reality glasses 100 of the augmented reality system 1, and Fig. 9A is an exemplary view of the augmented reality glasses 100. Fig.

9 and 9A, the augmented reality glasses 100 include a transparent display 110, a left front camera 121, a right front camera 122, a left 3D sensor 131, a right 3D sensor 131, Axis sensor 143, a battery 144, a recognition camera 145, and a control unit 150. The control unit 150 includes a control unit 132, a satellite module 141, a communication module 142, a nine-axis sensor 143,

The transparent display 110 constitutes a lens of the augmented reality glasses 100 as a transparent material display. Therefore, the user can simultaneously recognize the real object and the virtual object while looking ahead. At this time, the transparent display 110 may be mounted on the entire lens or a part of the lens.

The left front camera 121 is mounted on the left side of the glasses to acquire actual image information of the front side. Further, the right front camera 122 is mounted on the right side of the glasses to acquire the actual image information of the front side.

The left 3D sensor 131 and the right 3D sensor 132 operate in conjunction with the left front camera 121 and the right front camera 122 so as to take a 3D image of the front side. In other words, the photographed 3D image can be stored in the built-in memory or transmitted to the server 200. For reference, the front camera and the 3D sensor may be arranged one by one according to the embodiment to obtain actual image information. It is preferable that the front camera is configured to be able to photograph both the infrared region and the visible light region.

The communication module 142 may be equipped with a Wi-Fi communication module, a Bluetooth communication module, and a broadband (3G, 4G, LTE) communication module .

The 9-axis sensor 143 is referred to as a 9-axis sensor because the values of 9 axes, i.e., 3 axes of acceleration, 3 axes of inertia and 3 axes of geomagnetism are measured, and a temperature sensor may be additionally provided for correcting the temperature value . The 9-axis sensor 143 senses the three-dimensional motion of the augmented reality glasses 100 and can sense a gazing direction, a moving direction, a tilt, and the like of the user.

The battery 144 may be configured to be capable of supplying driving power to the augmented reality glasses 100 and may include a rechargeable lithium ion battery or a pseudo capacitor.

For reference, the battery 144 may be composed of a plurality of pseudo capacitors. The pseudo capacitors use a two-dimensional oxidation-reduction reaction at the electrodes, The advantage is that the life span is relatively long.

The recognition camera 145 detects the movement of the user's eyes, the gazing direction of the eyes, and the size of the eyes. The recognition camera 145 is most preferably disposed on the left and right sides, respectively, and may be disposed in only one direction.

Basically, although the recognition camera 145 is photographed in the direction in which the user's eyes are located, the recognition camera 145 may be configured to photograph an image of the eye reflected by the transparent display 110 to detect movement of the pupil, a gazing direction, will be.

The control unit 150 includes a transparent display 110, a left front camera 121, a right front camera 122, a left 3D sensor 131, a right 3D sensor 132, a satellite module 141, a communication module 142, The nine-axis sensor 143, the battery 144, and the recognition camera 145, as shown in FIG.

Meanwhile, the controller 150 may be configured to selectively charge at least one of the plurality of pseudo-capacitors according to the magnitude of the charging power. The charging method will be described in detail as follows.

It is assumed that when three pseudo-capacitors are arranged, that is, a first pseudo capacitor, a second pseudo capacitor, and a third pseudo capacitor are arranged. It is assumed that the charge capacity of the first pseudo capacitor is the largest, the charge capacity of the second pseudo capacitor is smaller than that of the first pseudo capacitor, and the charge capacity of the third pseudo capacitor is smaller than that of the second pseudo capacitor.

The controller 150 detects the charged amount of the first pseudo capacitor, the second pseudo capacitor, and the third pseudo capacitor, and then supplies the driving power in the order of the highest charge amount.

For example, when the charge amount of the first pseudo capacitor is 60%, the charge amount of the second pseudo capacitor is 70%, and the charge amount of the third pseudo capacitor is 80%

The power of the third pseudo capacitor is supplied first, and when the amount of charge reaches 40%, the power supply of the third pseudo capacitor is interrupted and the power of the second pseudo capacitor is supplied. Further, when the charged amount of the second pseudo capacitor reaches 40%, the power supply of the second pseudo capacitor is interrupted and the power of the first pseudo capacitor is supplied.

If the first to third pseudo-capacitors are all charged at 40% or less, the controller 150 connects the first to third pseudo capacitors in parallel to supply the driving power.

10 is a state diagram in which a safe mode of the augmented reality system 1 operates.

The augmented reality system 1 can be set to a safe mode for the user's safety.

When the safe mode is set, the augmented reality glasses 100 sense the actual object approaching the user through the front cameras 121 and 123. That is, it is possible to display a dangerous situation on the transparent display 110 by sensing that a real object, such as a car, a bicycle, etc., which is dangerous to the user, is approaching quickly to the user's direction.

Referring to FIG. 10, a screen in a state where a user is looking forward is shown. A rectangular area indicated by a dotted line in the center is defined as a sight line concentrated area, and the area is defined by a dotted line. In this case, when the front cameras 121 and 122 detect that the actual object, which may be a danger to the user such as a car or a bicycle, approaches the user in a fast direction (more than a predetermined speed)

The size of the sight line concentrated area is automatically expanded and the virtual object displayed on the screen automatically moves to the outward direction of the sight line concentrated area or the transparency is further strengthened so that the user can easily recognize the actual object accessed by the user do.

The size of the visual attention zone, the transparency of the virtual object, and the moving speed of the virtual object moving in the outward direction can be automatically determined in proportion to the speed of the actual object accessing the user (proportional to the direct proportion or square).

Further, when the recognition camera 145 detects the direction of the user's eyes, the eye-focused area operates to automatically move in accordance with the direction of the eyes. That is, when the eye of the user is gazing to the right, the eye-focused area is moved to the right direction. In this case, when the front cameras 121 and 123 detect that an actual object, which may be a danger to the user such as an automobile or a bicycle, is approaching from the front to the user, the sight line concentrated area may include a safety operation And the eye-focused area automatically moves in the front direction of the user.

That is, the eye-focused area may be set to automatically move in the direction of the actual object that is rapidly approaching the user.

In addition, a new virtual object may be assigned to a real object that may be a risk to the user, and a virtual line may be displayed to indicate the association with a physical object. At this time, a new virtual object may be displayed as an icon, a character indicating danger, and the access speed may be additionally displayed as a virtual object.

For reference, the recognition camera 145 can detect the movement of the user's pupil, the gaze direction of the pupil, and the change in the size of the eyes, and thus can instruct an operation command based on the change in size of the pupil.

For example, every time the user opens his or her eyes for a predetermined time, the virtual information corresponding to the lower information is gradually displayed, and the virtual information corresponding to the upper information is gradually displayed every time the eyes are opened for a predetermined time Can be instructed. Further, in order to improve the recognition rate of the recognition camera 145, an instruction eyebrow may be attached to a user's eyebrow. The pointing eyebrow may be coated with a reflective paint that reflects a predetermined infrared wavelength, and the recognition camera 145 may be configured to recognize the infrared wavelength to improve the command recognition rate.

11 is a state diagram in which the overhead view mode of the augmented reality system 1 operates.

Referring to FIG. 11, an augmented reality system 1 may have an overhead view mode.

The overhead view mode refers to a mode in which a composite image photographed on the user's head is displayed on the transparent display 110.

That is, although not shown in the figure, a plurality of view cameras capable of photographing the infrared ray region and the visible ray region may be additionally arranged along the eyeglass frames in the augmented reality glasses 100. Accordingly, the images photographed by the plurality of view cameras can be synthesized and provided to the user's field of view. In addition to the daytime, especially at night, the user can safely move the footprint. At this time, the height of the ground surface based on a predetermined previous position is displayed on the soles of the feet, which can help the user to move more safely.

In addition, the augmented reality system 1 may further include a plurality of sensing units 300, each of which includes a Wi-Fi communication module and is disposed at a predetermined interval in a room.

The plurality of sensing units 300 may be selectively arranged to detect the position of the augmented reality glasses 100 in the room,

Each of the plurality of sensing units 300 transmits sensing information to the server 200 every time it senses a WIFI HOTSPOT signal periodically output from the augmented reality glasses 100, The relative position of the augmented reality glasses 100 can be grasped based on the absolute position of the sensing unit 300.

As described above, in the proposed system, the present position information is acquired based on the satellite position information in the outdoor, and the current position information is acquired using the Wi-Fi signal in the room.

An additional method for acquiring current position information in indoor and outdoor is described as follows.

The method using the Wi-Fi signal can be basically divided into triangulation and fingerprinting.

First, triangulation measures the received signal strength (RSS) from three or more access points (APs), converts the signal intensity to distance, and calculates the position through the equation.

Next, in the fingerprinting method, the indoor space is divided into small cells, and signal intensity values are directly collected from each cell, and a database is constructed to form a radio map. Then, It estimates the cell that shows the most similar signal pattern compared with the database to the user's location.

Next, a method of collecting position data of each smartphone while exchanging Wi-Fi signals directly or indirectly from a plurality of users having smart phones in the vicinity may be used.

Also, since the communication module 142 of the augmented reality glasses 100 includes the Bluetooth communication module, the current position information can be grasped by using the Bluetooth communication.

First of all, there is a method of estimating that a user is located around a beacon when communicating with any one beacon after arranging a plurality of beacons in an indoor space.

Next, a receiver having a plurality of directional antennas arranged on the surface of a hemispherical shape is arranged in the indoor space, and then a position of a user is estimated through identification of a specific directional antenna that receives a signal transmitted from the augmented reality glasses 100 . In this case, when two or more receivers are arranged, the user's position may be identified in a three-dimensional form.

In addition, the current position information of the user is grasped based on the satellite position information, and the velocity and the moving direction are estimated using the information of the 9-axis sensor 143 when entering the shadow area of the satellite position information. That is, the position of the user can be estimated through indoor step counting, stride length estimation, and heading estimation. At this time, in order to increase the accuracy of the estimated information, information such as the user's physical condition (key, weight, stride) may be input and used for the position estimation calculation. When using the information of the 9-axis sensor 143, the accuracy of the estimated information can be improved by combining the position estimation technique using the Wi-Fi communication.

In other words, even if the accuracy is low with the Wi-Fi technique, the absolute position is globally calculated and the relative position of the locally high accuracy is obtained through the information of the 9- Can be combined to improve accuracy. In addition, the accuracy of the position estimation information may be further improved by further applying the Bluetooth communication method.

In addition, since the magnetic field information unique to the position is formed in the indoor space, a method of estimating the current position similarly to the fingerprinting method using Wi-Fi can be applied after constructing the magnetic field map of each space. At this time, the trend of the change in the magnetic field generated when the user moves can be used as additional information.

There is also a method of utilizing the illumination installed in the indoor space. That is, a method of outputting a specific position identifier while blinking the LED illumination at a speed which can not be recognized by a person, and the camera of the augmented reality glasses 100 recognizing the specific position identifier and estimating the position can be used.

In addition, there is a technique to database the images taken from various locations and various angles of the indoor space, and then to match the pictures taken at the user's location. In addition, various landmarks (signs, trademarks, room numbers, signs, etc.) A method of correcting the position can be used.

For reference, it is most preferable that the method of acquiring the current position information in the indoor space improves the accuracy by combining at least one of the above methods.

The augmented reality system 1 performs the role of the server 200 in the augmented reality glasses 100 itself without the configuration of the server 200 when the computing ability and the storage space of the augmented reality glasses 100 are sufficient .

The augmented reality system 1 according to the embodiment of the present invention can dynamically adjust the amount of information of the virtual object according to the distance between the object and the user so that the user can conveniently check information of the desired virtual object.

Therefore, when a virtual object is displayed in the form of an augmented reality advertisement, the time for the user to recognize the virtual object is concentrated, and the advertisement effect can be increased.

Also, the augmented reality system according to the embodiment of the present invention can select an additional recognition area and identify similar objects by assigning unique identifiers to real objects based on image differences in the additional recognition area.

Also, the augmented reality system can identify each real object by considering both the unique identifier assigned to each real object and the current position information of each real object based on the image difference of the additional recognition area.

Therefore, even if real objects having high similarity are arranged, it is possible to identify them and display the virtual objects allocated to the respective real objects, thereby conveying information without confusion to the user.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: augmented reality glasses
200: Server
300: sensing part
110: Transparent display
121: Left front camera
122: Right front camera
131: Left 3D sensor
132: Right 3D sensor
141: Satellite module
142: Communication module
143: 9 axis sensor
144: Battery
145: Recognition camera
150:

Claims (6)

A three-dimensional virtual image is displayed on a transparent display by capturing an image of a user's surroundings and acquiring actual image information. The three-dimensional virtual image corresponding to the current position information and the actual image information is displayed on the user's view Augmented reality glasses displayed within a range; And a server for providing the augmented reality glasses in real time with the current position information transmitted from the augmented reality glasses and the three-dimensional virtual image corresponding to the actual image information in real time, A method of providing a virtual object of a three-dimensional virtual image to an augmented reality eyeglass, the method comprising: identifying a plurality of objects; And a unique identifier is assigned to the real object based on the image difference of the additional recognition area,
Wherein the identification marker is printed on the additional recognition area, wherein the identification marker comprises: an identification marker area for reflecting the wavelength of the visible light area; And a plurality of identification marker regions each reflecting a different infrared wavelength, wherein the server identifies the identification marker through the recognition of each identification marker region and the relative position of each identification marker region, Identifying the actual object using the shadow position and size of each actual object as additional information based on the current position of the augmented reality glasses,
The augmented reality system further includes a plurality of sensing units, each of which includes a Wi-Fi communication module and is disposed at a predetermined interval in a room, and the plurality of sensing units include a WiFi hotspot periodically output from the augmented reality glasses, Transmits the detection information to the server every time it detects a signal,
Wherein a position of a virtual object of the three-dimensional virtual image allocated to each of the actual objects of the actual image information is automatically adjusted so as to maintain a predetermined distance from the actual object and displayed on the augmented reality glasses,
When a plurality of virtual objects are assigned to an entity object, the distance between the virtual objects is arranged closer to the virtual object, and the distance between the virtual objects is further automatically arranged when the association is lower.
Wherein the information amount of the virtual object is dynamically adjusted according to the distance between the actual object and the user and displayed on the augmented reality glasses.
The method according to claim 1,
The server comprises:
And identifies each real object by considering both the unique identifier given to each actual object and the current position information of each real object based on the image difference of the additional recognition area.
The method according to claim 1,
The server comprises:
And determines a candidate position of the additional recognition area based on current position information of each actual object.
The method according to claim 1,
The augmented reality glasses include:
And provides the satellite position information to the server as the current position information.
The method according to claim 1,
The augmented reality glasses include:
And transmits satellite position information to the server as the current position information, and further receives a signal intensity of at least one Wi-Fi repeater searched for and transmits the signal strength to the server.
delete

Images