KR101898075B1 - Augmented Reality System with Space and Object Recognition - Google Patents

Abstract

The present invention relates to an augmented reality system which accurately identifies an actual object even when the actual object partially exists in a capturing image and displays a virtual object assigned to the actual object at a targeted position. The augmented reality system simultaneously applied with space recognition and object recognition comprises an augmented reality terminal which is embedded with an image camera capturing an image around a user and obtaining actual image information and displays a three-dimensional virtual image on a display. The augmented reality terminal generates a space-based three-dimensional matching coordinate system by recognizing a space in three dimensions, displays each virtual object on a coordinate of the actual object individually assigned in advance based on the space-based three-dimensional matching coordinate system, updates the coordinate of the identified actual object after identifying the coordinate of the actual object again whenever the actual object of the actual image information is image-identified based on object recognition, and displays the virtual object previously assigned to the coordinate of the updated actual object.

Description

{Augmented Reality System with Space and Object Recognition}

Field of the Invention [0002] The present invention relates to an augmented reality system, and more particularly, to an augmented reality system capable of displaying a position of a virtual object at an accurate position by simultaneously applying space recognition and object recognition.

Recently, contents are being actively researched using augmented reality technique in which various information is superimposed on a real space when photographing using a camera module.

Augmented Reality (AR) is a technology belonging to one of the fields of Virtual Reality (VR). It combines a virtual environment with a real environment that a user feels in a sense, and feels as if a virtual environment exists in the original real environment Is a computer technique.

This augmented reality is advantageous in that it can supplement and supplement additional information that is difficult to obtain only in the real world by synthesizing virtual objects on the basis of the real world, unlike the existing virtual reality which only targets virtual space and objects.

That is, a realization realized by matching a virtual environment created by computer graphics to a real image viewed by a user, for example, a three-dimensional virtual environment, can be called an augmented reality. Here, the 3D virtual environment provides necessary information from the real image that the user views, and the 3D virtual image matches with the real image to increase the user's immersion.

Such augmented reality can provide better realism by making the distinction between the real environment and the virtual environment be blurred by providing the real image in the three-dimensional virtual environment as well as the simple virtual reality technology.

Due to the advantages of such an augmented reality system, active research and development are currently under way for technologies that have been combined with augmented reality around the world. For example, augmented reality technology is being put into practical use in fields such as broadcast, advertisement, exhibition, game, theme park, military, education and promotion.

Recently, the handheld augmented reality system has become possible by improving the computing ability of mobile devices such as mobile phones, personal digital assistants (PDAs), and ultra mobile personal computers (UMPCs) and wireless network devices.

As these systems become available, many augmented reality applications using mobile devices have been developed. In addition, the spread of mobile devices is becoming very common and an environment for accessing augmented reality applications is being created.

In addition, user demands for various supplementary services using an augmented reality of a terminal are increasing, and attempts to apply various augmented reality contents to a terminal user are increasing.

Korean Patent Laid-Open Publication No. 10-2016-0092292 relates to a system and method for providing an augmented reality service for an advertisement, and it is an object of the present invention to apply an augmented reality to an advertisement, And the system is able to communicate effectively by being intrigued and intriguing.

Meanwhile, various wearable device products that can be worn on the body of the user are being released. In particular, a system has been developed in which mobile technology and transparent display technology are applied to glasses to allow users to check various information through glasses.

1 is a diagram illustrating a virtual object displayed in a conventional augmented reality system.

Referring to FIG. 1, the conventional augmented reality system displays too many virtual objects in a transparent display of eyeglasses, thereby hindering the user's eyesight. For this reason, it is difficult for the user to recognize necessary information.

Also, there is a need for a technique that can accurately identify virtual objects existing in a captured image or exist partially or partially, and display a virtual object allocated to a real object at a desired position.

In addition, if there are a plurality of actual objects having similar appearance, duplicate recognition or unassigned virtual objects may be displayed to cause confusion, so that a technology capable of identifying duplicate objects and preventing duplicate recognition is required.

KR 10-2016-0092292 A

SUMMARY OF THE INVENTION It is an object of the present invention to provide an augmented reality system capable of accurately identifying a virtual object even if the actual object partially exists in the captured image and displaying the virtual object allocated to the real object at a desired position to provide.

According to an aspect of the present invention, there is provided an augmented reality terminal including an image camera for capturing an image of a user around the user and acquiring real image information, and displaying a three-dimensional virtual image on a display, The augmented reality terminal recognizes the space three-dimensionally to generate a space-based three-dimensional matching coordinate system, and displays each virtual object in the coordinates of the real objects pre-allocated based on the space-based three-dimensional matching coordinate system The coordinates of the actual object are grasped again each time the actual object of the actual image information is image-recognized based on object recognition, the coordinates of the actual object are updated, and the coordinates of the actual object, And a virtual object is displayed on the display device.

According to another embodiment of the present invention, there is provided an augmented reality terminal, which includes a video camera for capturing an image of a user and acquiring actual image information, and displaying a three-dimensional virtual image on a display, Dimensional space, and generates a space-based three-dimensional matching coordinate system, and displays each virtual object in the coordinates of the real objects allocated in advance based on the space-based three-dimensional matching coordinate system, Dimensional matching coordinate system based on the object recognition based on the coordinates of the actual object each time the actual object of information is image-recognized based on the object recognition based on the three-dimensional matching coordinate system based on the object recognition, Based three-dimensional matching coordinate system is updated.

According to another embodiment of the present invention, there is provided an image display apparatus including a video camera for capturing an image of a user and acquiring real image information, and displaying a three-dimensional virtual image on a display, And displaying the 3D virtual image corresponding to the current image information and the current position information transmitted from the augmented reality terminal in the visual range of the user, Wherein the server provides the virtual object of the 3D virtual image assigned to the identified real object to the augmented reality terminal by image identification of the actual object of the actual image information, The real terminal recognizes the space three-dimensionally and generates a space-based 3D matching coordinate system Wherein each virtual object is displayed in the coordinates of an actual object previously allocated based on the space-based three-dimensional matching coordinate system, and each time the actual object of the actual image information is image- The coordinates of the actual object are updated, and the virtual object previously allocated to the coordinates of the updated actual object is displayed.

According to another embodiment of the present invention, there is provided an image display apparatus including a video camera for capturing an image of a user and acquiring real image information, and displaying a three-dimensional virtual image on a display, And displaying the 3D virtual image corresponding to the current image information and the current position information transmitted from the augmented reality terminal in the visual range of the user, To a server,

Wherein the server provides the virtual object of the 3D virtual image assigned to the identified real object to the augmented reality terminal by image identification of the actual object of the actual image information, Based three-dimensional matching coordinate system, and displays each virtual object in the coordinates of the real object allocated in advance based on the space-based three-dimensional matching coordinate system, wherein the actual object of the actual image information is Dimensional coordinate system based on the object recognition, after recognizing the coordinates of the actual object each time the image is identified based on the object recognition, and generating the three-dimensional coordinate system based on the object recognition based on the three- And the coordinate system is updated.

The augmented reality system according to the embodiment of the present invention can correctly identify the actual object even if it does not exist in the captured image or partially exist and display the virtual object allocated to the actual object at the target position.

That is, it is possible to display the position of the virtual object at the correct position by simultaneously applying the space recognition and the object recognition.

1 is a diagram showing a virtual object displayed in a conventional augmented reality system;
2 is a block diagram of an augmented reality system 1 according to an embodiment of the present invention.
3 is a view showing an augmented reality realized on the basis of object recognition
3A is a diagram showing an augmented reality realized based on spatial recognition;
4 is a conceptual diagram of the operation of the augmented reality system 1. Fig.
5 is an exemplary diagram showing an operation state of the augmented reality system 1. Fig.
6 is a flowchart showing a learning process for identifying similar objects in the augmented reality system 1. Fig.
6A is a diagram showing a process of selecting an additional recognition area for identifying a similar object in the AR system 1
7 is a flowchart showing a process of identifying a similar object in the augmented reality system 1
8 is a first exemplary diagram showing a state for identifying a similar object in the augmented reality system 1. Fig.
9 is a second example showing a state in which similar objects are identified in the augmented reality system 1. Fig.
10 is a diagram showing another operation principle of the augmented reality system 1
11 and 11A are diagrams showing still another operation principle of the augmented reality system 1
12 is a configuration diagram of the augmented reality terminal 100 of the augmented reality system 1. Fig.
12A is an exemplary diagram of the augmented reality terminal 100
13 is a state diagram in which the safe mode of the augmented reality system 1 is operated
14 is a diagram showing a state in which the overhead view mode of the augmented reality system 1 is operated

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, in order to facilitate a person skilled in the art to easily carry out the technical idea of the present invention.

2 is a configuration diagram of an augmented reality system 1 according to an embodiment of the present invention.

The augmented reality system 1 according to the present embodiment includes only a brief configuration for clearly explaining the technical idea to be proposed.

2, the augmented reality system 1 includes an augmented reality terminal 100, a server 200, and a plurality of sensing units 300. As shown in Fig. Here, the plurality of sensing units 300 or the server 200 may be selectively provided in the augmented reality system 1 according to the embodiment.

The augmented reality system 1 can play the role of the server 200 in the augmented reality terminal 100 itself without the configuration of the server 200 when the computing ability and the storage space of the augmented reality terminal 100 are sufficient As shown in FIG.

In addition, in the present embodiment, the augmented reality terminal 100 may be composed of augmented reality glasses or a mobile terminal. The detailed configuration of the augmented reality glasses will be described later, and the mobile terminal is collectively referred to as a mobile phone, a smart phone, a smart pad, etc., which can be used while the user is carrying it. .

Basically, the augmented reality terminal 100 has a built-in video camera for capturing an image of a user and acquiring real image information, and displaying a three-dimensional virtual image on a display, The virtual image is displayed within the visual range of the user.

The server 200 provides a three-dimensional virtual image corresponding to the current location information and the actual image information transmitted from the augmented reality terminal 100 to the augmented reality terminal 100 in real time.

The augmented reality terminal 100 is basically configured to provide the satellite position information to the server 200 as current position information. When augmented reality terminal 100 includes a communication module, not only the satellite position information but also the location of a nearby Wi-Fi repeater, the location of a base station, and the like can be additionally provided to the server 200 as current location information have.

The augmented reality terminal 100 may further determine the signal strength of at least one of the searched Wi-Fi repeaters and transmit the satellite position information to the server 200, . That is, since the absolute position of the Wi-Fi repeater located indoors is stored in advance in the server 200, the augmented reality terminal 100 additionally provides the unique number and signal strength of the Wi-Fi repeater searched for , The server 200 can grasp the relative movement path of the augmented reality terminal 100.

That is, the relative distance between the augmented reality terminal 100 and the Wi-Fi repeater can be confirmed by the signal strength, and the direction of movement can be calculated based on the change in the signal intensity with the neighboring Wi-Fi repeater . An additional method for acquiring the current position information in the room will be described later.

Therefore, the server 200 can acquire the virtual object (s) assigned to the actual object through the current position information of the user wearing or holding the augmented reality terminal 100 and the actual image information photographed by the image camera of the augmented reality terminal 100. [ And transmits the information to the augmented reality terminal 100 in real time.

Meanwhile, there are two methods of identifying actual objects in the room.

FIG. 3 is a diagram illustrating an augmented reality based on object recognition, and FIG. 3A is a view illustrating an augmented reality implemented based on a spatial recognition.

Referring to FIGS. 3 and 3A, the method of identifying an actual object on the basis of object recognition is easy because the size of an object is small compared to a space. However, if objects are included in the image, recognition can be performed, and restrictions on the recognition direction and distance may occur. Storing a location in advance is a difficult task because an object can change its location. As a reference, one object matching object coordinate system (augmented reality matching coordinate system) is generated for each object.

Next, SLAM (Simultaneous Localization and Map-Building, Simultaneous Localization and Mapping) technology is applied to the method of identifying the location of the actual object based on the spatial recognition. SLAM is a task to create a precise 3D map of the environment without the help of the outside only by the sensor attached to the terminal (robot) while the terminal (robot) travels around the unknown environment. It is defined as the core technology for autonomous driving .

In other words, the space recognition based method is a method of creating a three-dimensional matching coordinate system of space by three-dimensionally recognizing a space and displaying each virtual object in coordinates of an actual object allocated in advance according to a three- dimensional matching coordinate system .

Therefore, a virtual object can be augmented on an object (real object) even if the object is not included in the image. However, when the terminal is moved from the initial recognition point, the spatial tracking error may accumulate and it may be difficult for the augmented reality information (virtual object) to be displayed at an appropriate position. For reference, one space-based three-dimensional matching coordinate system (augmented reality matching coordinate system) is generated per space.

Accordingly, the augmented reality system 1 according to the present invention can accurately identify virtual objects, which are all present in the photographed image, all of them not present or partially existing, and displaying the virtual objects allocated to the actual objects at the target positions So that the position of the virtual object can be displayed at the correct position by simultaneously applying the space recognition and the object recognition.

That is, the augmented reality terminal 100 recognizes the space three-dimensionally to generate a space-based three-dimensional matching coordinate system, and assigns each virtual Dimensional object coordinate system based on object recognition after recognizing the coordinates of the actual object each time the actual object of the actual image information is image-recognized based on the object recognition based on the three-dimensional coordinate system based on the object recognition, Based three-dimensional matching coordinate system is updated based on the three-dimensional coordinate system.

4 is a conceptual diagram of the operation of the augmented reality system 1, and Fig. 5 is an exemplary view showing the operation state of the augmented reality system 1. As shown in Fig.

The operation principle of the augmented reality system 1 will be described with reference to FIGS. 4 and 5. FIG.

The augmented reality terminal 100 recognizes the space three-dimensionally, generates a space-based three-dimensional matching coordinate system, and assigns each virtual object to the coordinate of the actual object allocated in advance based on the space-based three-dimensional matching coordinate system Display.

Therefore, even if the entity object does not exist in the photographed image or partially exists, the virtual object allocated to the actual object can be displayed at the target position using only the space-based 3D matching coordinate system.

At this time, each time the actual object of the actual image information is image-recognized based on the object recognition, the augmented reality terminal 100 grasps the coordinates of the actual object, updates the coordinates of the detected actual object, The virtual object may be displayed in advance. That is, basically, it is possible to identify the location of the entity object based on the spatial recognition, and simultaneously update the coordinates of the location of the identified actual object after applying the object recognition basis.

On the other hand, the augmented reality terminal 100 recognizes the space three-dimensionally to generate a space-based three-dimensional matching coordinate system, and assigns each virtual object Lt; / RTI >

At this time, each time the actual object of the actual image information is image-recognized based on the object recognition, the augmented reality terminal 100 obtains the coordinates of the actual object, and further generates a three-dimensional matching coordinate system based on the object recognition. Based three-dimensional matching coordinate system based on the dimensional matching coordinate system.

That is, each time an actual object is identified based on object recognition, a space-based 3-dimensional matching coordinate system can be estimated and corrected based on the coordinates of the actual object.

As shown in FIG. 5, each time an object (real object) is image-recognized based on object recognition, a three-dimensional matching coordinate system #n based on object recognition is additionally generated, and based on this, The accumulated error of the space-based three-dimensional matching coordinate system due to the movement of the augmented reality terminal 100 or the like can be corrected.

In this case, when a plurality of objects (real objects) are recognized based on object recognition, the augmented reality terminal 100 can sense a change in the position of an actual object in consideration of a relative position between the actual objects.

Therefore, the augmented reality terminal 100 not only can confirm whether the actual object is moved through the space-based three-dimensional matching coordinate system, but also detects the change in the position of the actual object by further considering the relative position between the actual objects It will be possible.

The augmented reality system 1 performs three-dimensional spatial location recognition (SLAM algorithm) while at the same time recognizing / tracking objects for parallel interaction and matching contents.

First, the augmented reality terminal 100 generates an augmented reality matching coordinate system by performing three-dimensional space learning (SLAM algorithm). Next, the attitude (movement, rotation) of the terminal is tracked based on the co-ordinate coordinate system of the space. If the previously learned space is recognized, the augmented reality matching coordinate system defined in advance can be called.

Next, the augmented reality terminal 100 tries to recognize an object from a camera image, and when the object is recognized, a three-dimensional matching coordinate system based on object recognition is additionally generated. Next, the information (content) is augmented on the object. Even if a large part of the object is not visible in the camera image of the augmented reality terminal 100, the virtual information can be enhanced based on the space and the object.

In summary, when the augmented reality system 1 comprises the augmented reality terminal 100 and the server 200,

The server 200 images the real objects of the real image information and provides the virtual objects of the three-dimensional virtual images assigned to the identified real objects to the augmented reality terminal 100,

The augmented reality terminal 100 recognizes the space three-dimensionally, generates a space-based three-dimensional matching coordinate system, and assigns each virtual object to the coordinate of the actual object allocated in advance based on the space-based three-dimensional matching coordinate system The coordinates of the actual object are reassembled every time the actual object of the actual image information is image-recognized based on object recognition, the coordinates of the actual object are updated, and the coordinates of the updated real object are updated. .

In addition, the augmented reality terminal 100 recognizes spaces three-dimensionally to generate a space-based three-dimensional matching coordinate system, and assigns each virtual Dimensional object coordinate system based on object recognition after recognizing the coordinates of the actual object each time the actual object of the actual image information is image-recognized based on the object recognition based on the three-dimensional coordinate system based on the object recognition, Based three-dimensional matching coordinate system is updated based on the three-dimensional coordinate system.

Meanwhile, the augmented reality system 1 according to the embodiment of the present invention can identify the similar object by selecting an additional recognition area and assigning a unique identifier to the actual object based on the image difference of the additional recognition area.

In addition, the augmented reality system 1 can identify each real object by considering both the unique identifier given to each real object and the current position information of each real object based on the image difference in the additional recognition area.

Therefore, even if real objects having high similarity are arranged, it is possible to identify them and display the virtual objects allocated to the respective real objects, thereby conveying information without confusion to the user.

The visual cognition description of an object can be divided into four steps. - DCRI (Detection, Classification, Recognition, and Identification) -

First, detection is a step that can detect whether or not an object exists.

Next, Classification is a step of knowing what classification an object is. - For example, it can be classified as human or animal -

Next, Recognition is a step to know the approximate characteristics of the object. - For example, you can find brief information about clothes that a person wears -

Finally, Identification is a step that allows to know the detailed characteristics of an object. For example, you can identify a person's face and know the number of the license plate.

The augmented reality system 1 of the present invention operates so that an identification step can be implemented to distinguish detailed characteristics of similar real objects.

For example, the augmented reality system 1 recognizes a character attached to a facility (real object) having a similar shape and assigns a unique identification number, or identifies a difference portion, and then selects the additional recognition region The 2D / 3D feature information difference of the additional recognition area, and the current position information measured by the satellite position information and the Wi-Fi signal, to distinguish real objects having high similarity values.

FIG. 6A is a flowchart illustrating a learning process for identifying similar objects in the AR system 1, FIG. 6A is a flowchart illustrating a process of selecting an additional recognition area for identifying similar objects in the AR system 1 .

6 and 6A, the server 200 operates to identify a real object of actual image information and to provide a virtual object of a three-dimensional virtual image allocated to each identified real object to the augmented reality terminal 100 .

That is, the server 200 selects the additional recognition area by subtracting the actual image information from the actual objects having the predetermined image similarity value d or more among the plurality of actual objects existing in the actual image information, And assigns a unique identifier to the actual object based on the image difference of the object.

For example, when characters or numbers different from each other are displayed in the additional recognition area, the server 200 assigns a unique identifier to each of the actual objects based on the difference of the additional recognition areas, stores the database in a database, And can transmit the assigned virtual object to the augmented reality terminal 100.

That is, when a plurality of actual objects have a predetermined image similarity value (d) or more, the image is abstracted and the image is differentiated to set an additional recognition area (additional learning area) And each unique identifier is assigned.

FIG. 7 is a flowchart illustrating a process of identifying similar objects in the AR system 1, and FIG. 8 is a first exemplary view illustrating a state in which similar objects are identified in the AR system 1. FIG.

7 and 8, when the actual image information is transmitted from the augmented reality terminal 100, the server 200 identifies the actual object of the actual image information, and if the similar image does not exist If there is no actual object), virtual objects corresponding to the identified image are respectively allocated.

At this time, the server 200 compares the information of the additional recognition area with the similar image (if a similar object exists), identifies the unique identifier, and then assigns the virtual object corresponding to the unique identifier.

8, when similar facilities (actual objects) are arranged at neighboring positions, the server 200 may extract a plurality of actual objects from actual image information transmitted from the augmented reality terminal 100 Then, the information of the additional recognition area of each real object is compared to identify the unique identifier, and the virtual object corresponding to the unique identifier is allocated.

On the other hand, when different identification markers are printed in the additional recognition area of each facility, the shape of the identification markers can be configured as follows.

The identification marker may be configured to include a first identification marker region, a second identification marker region, a third identification marker region, and a fourth identification marker region.

That is, the identification marker recognizes the first identification marker region, the second identification marker region, the third identification marker region, and the fourth identification marker region as one identifier. That is, the augmented reality terminal 100 basically captures all the first to fourth identification markers and transmits them to the server 200, and the server 200 recognizes the recognized identification markers as one unique identifier.

At this time, the first identification marker region is configured to reflect the wavelength of the visible light region. That is, the first identification marker area is printed with a common paint and printed so that a human can visually distinguish it.

In addition, the second identification marker region reflects the first infrared wavelength, printed with a paint reflecting the first infrared wavelength, and can not be visually distinguished by a person.

Further, the third identification marker region reflects a second infrared wavelength of a wavelength longer than the first infrared wavelength, and is printed with a paint reflecting the second infrared wavelength, and can not be visually distinguished by a person.

Further, the fourth identification marker region simultaneously reflects the first infrared wavelength and the second infrared wavelength, which is printed with a paint reflecting the first and second infrared wavelengths, and can not be visually distinguished by a person.

At this time, the camera of the augmented reality terminal 100 for imaging the identification marker is configured to be equipped with a spectral filter for controlling the infrared transmission wavelength so as to photograph and recognize the infrared wavelength region.

Therefore, only the first identification marker region of the identification markers printed on the facility can be visually confirmed by the person, and the second identification marker region, the third identification marker region and the fourth identification marker region can not be visually confirmed by the person, And can be photographed through the camera of the terminal 100.

Relative printing positions (left, right, top and bottom) of the first identification marker region, the second identification marker region, the third identification marker region and the fourth identification marker region can also be used as one delimiter. In the identification marker area, various characters such as numbers, symbols, and symbols may be printed. In addition, the identification marker may be printed in the form of a QR code or a bar code.

9 is a second exemplary view showing a state in which similar objects are identified in the augmented reality system 1. Fig.

Referring to FIG. 9, the server 200 can identify each real object by considering both the unique identifier assigned to each real object and the current position information of each real object based on the image difference in the additional recognition area . That is, the actual object can be identified by further considering the current location information of the user (the augmented reality terminal 100).

For example, when a plurality of actual objects maintain a predetermined distance, the user can identify the object using the current location information even if the actual object has a high similarity value. Herein, it is assumed that the current position information includes the absolute position information of the user, relative position information, the moving direction, the acceleration, and the direction of the line of sight.

At this time, the server 200 may select an additional recognition area to further identify actual objects not identified by the current location information, and recognize the difference to identify differences between actual objects.

Also, the server 200 can determine the candidate position of the additional recognition area based on the current position information of each actual object.

That is, referring to FIG. 9, when the user gazes at the actual object located at the first position P1 and located at the front,

The server 200 determines the distance between the real object and the user based on the actual image information, and then detects the coordinates (x1, y1, z1) on the space of the actual object.

Since the plurality of additional recognition areas are previously set in the actual object located at the coordinates (x1, y1, z1) on the space, the server 200 stores the coordinates (x1, y1, z1) The candidate position of the additional recognition area can be determined based on the current position information.

The server 200 grasps in advance which real object exists in the coordinates (x1, y1, z1) on the space and which part of the actual object is designated as the additional recognition area, It is possible to identify an object by identifying only a candidate position. Such a scheme has an advantage that the amount of calculation for identifying an additional recognition area can be reduced.

For reference, assuming indoor space and assuming that all directions of illumination are constant, the position and size of shadows due to the illumination may be different from each other even for similar real objects. Accordingly, the server 200 may identify each actual object using the difference between the shadow position and the size of each actual object as additional information based on the current position of the user.

On the other hand, the positions of the virtual objects of the three-dimensional virtual images allocated to the actual objects of the actual image information are automatically adjusted to maintain a predetermined distance from the actual objects and displayed on the augmented reality terminal 100.

In addition, the positions between the virtual objects are automatically adjusted to maintain a predetermined separation distance, and displayed on the augmented reality terminal 100.

Therefore, since the position is automatically adjusted so that the objects are not overlapped with each other considering the mutual positional relationship between the actual object and the virtual object, the user can conveniently check the information of the desired virtual object. That is, it takes a long time to recognize the virtual objects with concentration, and the advertisement effect can be increased.

10 is a view showing another operation principle of the augmented reality system 1. Fig.

The operation principle of the augmented reality system 1 will be described in more detail with reference to FIG.

The distance D2 between the first virtual object (Virtual Object 1) and the actual object (Physical Object)

The distance R1 between the center point of the first virtual object 1 and the outermost area of the first virtual object 1 and the distance R1 between the center point of the actual object and the center of the physical object Is set to be longer than the sum of the distance R3 between the outer areas.

The distance Dl between the second virtual object 2 and the physical object is a distance between the center point of the second virtual object 2 and the center of the second virtual object 2 The distance R2 between the outer regions and the distance R3 between the center point of the physical object and the outermost region of the physical object.

The distance D1 between the first virtual object 1 and the second virtual object 2 is calculated by dividing the distance between the center point of the first virtual object 1 and the first virtual object 1 And a distance R2 between the center point of the second virtual object 2 and the outermost region of the second virtual object 2 (Virtual Object 2), which is longer than the sum of the distance R1 between the outermost region of the second virtual object .

The first virtual object (Virtual Object 1) and the second virtual object (Virtual Object 2) move in the horizontal direction and the vertical direction and are automatically overlapped with other objects to automatically adjust the position in the three-dimensional space so as not to deviate from the user's view. , the position is adjusted with respect to the y and z axes.

On the other hand, between the physical object and the first virtual object 1, and between the physical object and the second virtual object 2, the virtual lines L1 and L2 are dynamic As shown in FIG.

When a large number of virtual objects exist on the screen, the virtual lines L1 and L2 are displayed to indicate the association with the physical object, and the thickness, transparency, and color of the virtual lines L1 and L2 are It can be changed automatically according to the line of sight.

For example, when the user gazes at the first virtual object (Virtual Object 1) for a predetermined time or longer, the augmented reality terminal 100 detects whether or not the first virtual object 1 is examined, The thickness of the virtual line L1 between the first virtual object 1 and the first virtual object 1 is changed to be thicker than the other virtual line L2 and the transparency is changed to be lower, It may also operate to automatically change to color.

At this time, assuming that the first virtual object 1 and the second virtual object 2 are all allocated to the physical object, the distance of the plurality of virtual objects allocated to the physical object is As described above, virtual objects having more detailed information are held closer to a physical object while maintaining a predetermined distance D2 and D3, respectively.

For example, if the information of the first virtual object (Virtual Object 1) is more detailed information and the information of the second virtual object (Virtual Object 2) is relatively conceptual information,

The distance D3 between the second virtual object 2 and the physical object is longer than the distance D2 between the first virtual object 1 and the physical object It is automatically set up so that the user can quickly recognize detailed information.

Also, when a plurality of virtual objects are allocated to a physical object, the distance between the virtual objects is arranged closer to the virtual object, and the distance between the virtual objects is arranged more automatically have.

11 and 11A are diagrams showing still another operation principle of the augmented reality system 1. Fig.

11 and 11A, the amount of information of a virtual object of a three-dimensional virtual image allocated and displayed to real objects of the actual image information is automatically dynamically adjusted according to the distance between the actual object and the user, 100 < / RTI >

Therefore, when the user approaches the object, the virtual object displays more detailed information, so that the user can conveniently check the information of the desired virtual object. That is, the time for the user to concentrate on the virtual object is prolonged, and the information transfer effect can be increased.

In general, the user tends to look closer at the information about the object of interest, so that when the user is far away from the object of interest, the information is abstractly displayed, and when the user is close to the object of interest, .

A virtual object assigned to an actual object starts to be displayed from a time when a distance between a user and a real object or a virtual object reaches within a predetermined separation distance D1 and the closer the distance between the user and the actual or virtual object becomes A virtual object of more detailed information is displayed.

That is, information of a virtual object allocated to one real object is layered. As shown in FIG. 8A,

The most abstract virtual object A is displayed when the user enters a predetermined separation distance D1 and becomes a virtual object having more specific information when the user approaches (D2) the actual or virtual object more A1, A2) are displayed. Also, when the user approaches (D3) closest to the actual object or the virtual object, the virtual objects (A1-1, A1-2, A2-1, A2-2) having the most specific information are displayed.

For example, assuming that a vending machine is placed as a real object in front of the user,

When the user enters the predetermined separation distance D1, the virtual object allocated to the vending machine is displayed. Here, the virtual object is assumed to be the icon of the vending machine.

Next, when the user approaches the vending machine more (D2), the icons of the vending products sold in the vending machine can be displayed as virtual objects of more detailed information.

Finally, when the user closest approaches the vending machine (D3), the calories, ingredients, etc. of each beverage product can be displayed as a virtual object of more detailed information.

As another example, assuming that there is an automobile store as a real object in front of the user,

When the user enters the predetermined separation distance D1, the virtual object allocated to the car dealership is displayed. Here, the virtual object is assumed to be a car brand icon to be sold.

Next, when the user accesses the car dealer more (D2), the various kinds of car icons being sold can be displayed as virtual objects of more detailed information. At this time, a virtual object can be displayed in the vicinity of the currently displayed car, that is, an actual object if there is an actual object, and a virtual object can be displayed in the surrounding area even when the car is not displayed .

Finally, when the user closest approach (D3) to the car dealership, the specification of the vehicle being sold, the price, and the expected date of shipment can be displayed as virtual objects of more detailed information.

Meanwhile, when a vibration occurs in a state where a user looks at a virtual object or an actual object that the user wants to check, a movement distance corresponding to a rate of change of vibration is calculated, and then a gaze direction and a calculated movement distance are calculated The amount of information of the virtual object can be reset and displayed.

That is, it is assumed that the user has moved by virtual vibration without directly moving, or a weight is given to the movement distance by vibration, and the information amount of the virtual object is reset according to the virtual movement distance and displayed.

That is, the large change rate of the vibration means that the user is running or moving fast, and the small change rate of the vibration corresponds to the user moving slowly, so that the travel distance can be calculated based on this. Therefore, the user may be able to reflect the virtual movement distance by imparting vibration while shaking the head up and down without actually moving.

When the user gazes at a direction to be checked by the user, for example, when the user turns his or her head to the right and continues to vibrate while looking at the right side, the augmented reality terminal 100, based on the gazing direction and the vibration of the user, And the movement distance.

That is, the augmented reality terminal 100 detects the head rotation of the user through the built-in sensor, calculates the virtual current position after detecting the vibration, and determines the movement distance of the walking or running behavior through the rate of change of vibration .

The augmented reality terminal 100 may be configured to detect the gazing direction based on the direction of rotation of the head when detecting the gazing direction of the user and to sense the gazing direction by sensing the moving direction of the pupil .

Further, it is possible to simultaneously detect the rotational direction of the head and the moving direction of the pupil, and calculate the gaze direction more precisely by differentiating the specific gravity first from the two detection results. That is, it is possible to provide a specific gravity of 50% to 100% for detecting the rotation angle due to rotation of the head, and to calculate the gaze direction by giving a specific gravity of 0% to 60% to the moving direction of the pupil.

Also, the user can set the moving distance extension setting mode in the augmented reality terminal 100 so that the distance weight is 2 to 100 times the calculated virtual moving distance.

In order to exclude the noise value, the augmented reality terminal 100 may exclude the upper value 10% of the vibration magnitude and the lower value 20% and calculate the change rate of the remaining vibration value It is possible to calculate the moving distance.

As a result, the user can identify a virtual object that has the same amount of information as a real object or a virtual object, even if the user does not actually access the actual object or the virtual object or approaches the virtual object.

12 is a configuration diagram of the augmented reality terminal 100 of the augmented reality system 1, and Fig. 12A is an exemplary view of the augmented reality terminal 100. As shown in Fig.

12 and 12A, the augmented reality terminal 100 includes a transparent display 110, a left front camera 121, a right front camera 122, a left 3D sensor 131, a right 3D sensor 131, Axis sensor 143, a battery 144, a recognition camera 145, and a control unit 150. The control unit 150 includes a control unit 132, a satellite module 141, a communication module 142, a nine-axis sensor 143,

The transparent display 110 is a transparent material and constitutes a lens of the augmented reality terminal 100. Therefore, the user can simultaneously recognize the real object and the virtual object while looking ahead. At this time, the transparent display 110 may be mounted on the entire lens or a part of the lens.

The left front camera 121 is mounted on the left side of the glasses to acquire actual image information of the front side. Further, the right front camera 122 is mounted on the right side of the glasses to acquire the actual image information of the front side.

The left 3D sensor 131 and the right 3D sensor 132 operate in conjunction with the left front camera 121 and the right front camera 122 so as to take a 3D image of the front side. In other words, the photographed 3D image can be stored in the built-in memory or transmitted to the server 200. For reference, the front camera and the 3D sensor may be arranged one by one according to the embodiment to obtain actual image information. It is preferable that the front camera is configured to be able to photograph both the infrared region and the visible light region.

The communication module 142 may be equipped with a Wi-Fi communication module, a Bluetooth communication module, and a broadband (3G, 4G, LTE) communication module .

The 9-axis sensor 143 is referred to as a 9-axis sensor because the values of 9 axes, i.e., 3 axes of acceleration, 3 axes of inertia and 3 axes of geomagnetism are measured, and a temperature sensor may be additionally provided for correcting the temperature value . The 9-axis sensor 143 senses the three-dimensional movement of the augmented reality terminal 100, and can sense a gazing direction, a moving direction, a tilt, and the like of the user.

The battery 144 may be configured to be capable of supplying driving power to the augmented reality terminal 100, and may include a rechargeable lithium ion battery or a pseudo capacitor.

For reference, the battery 144 may be composed of a plurality of pseudo capacitors. The pseudo capacitors use a two-dimensional oxidation-reduction reaction at the electrodes, The advantage is that the life span is relatively long.

The recognition camera 145 detects the movement of the user's eyes, the gazing direction of the eyes, and the size of the eyes. The recognition camera 145 is most preferably disposed on the left and right sides, respectively, and may be disposed in only one direction.

Basically, although the recognition camera 145 is photographed in the direction in which the user's eyes are located, the recognition camera 145 may be configured to photograph an image of the eye reflected by the transparent display 110 to detect movement of the pupil, a gazing direction, will be.

The control unit 150 includes a transparent display 110, a left front camera 121, a right front camera 122, a left 3D sensor 131, a right 3D sensor 132, a satellite module 141, a communication module 142, The nine-axis sensor 143, the battery 144, and the recognition camera 145, as shown in FIG.

Meanwhile, the controller 150 may be configured to selectively charge at least one of the plurality of pseudo-capacitors according to the magnitude of the charging power. The charging method will be described in detail as follows.

It is assumed that when three pseudo-capacitors are arranged, that is, a first pseudo capacitor, a second pseudo capacitor, and a third pseudo capacitor are arranged. It is assumed that the charge capacity of the first pseudo capacitor is the largest, the charge capacity of the second pseudo capacitor is smaller than that of the first pseudo capacitor, and the charge capacity of the third pseudo capacitor is smaller than that of the second pseudo capacitor.

The controller 150 detects the charged amount of the first pseudo capacitor, the second pseudo capacitor, and the third pseudo capacitor, and then supplies the driving power in the order of the highest charge amount.

For example, when the charge amount of the first pseudo capacitor is 60%, the charge amount of the second pseudo capacitor is 70%, and the charge amount of the third pseudo capacitor is 80%

The power of the third pseudo capacitor is supplied first, and when the amount of charge reaches 40%, the power supply of the third pseudo capacitor is interrupted and the power of the second pseudo capacitor is supplied. Further, when the charged amount of the second pseudo capacitor reaches 40%, the power supply of the second pseudo capacitor is interrupted and the power of the first pseudo capacitor is supplied.

If the first to third pseudo-capacitors are all charged at 40% or less, the controller 150 connects the first to third pseudo capacitors in parallel to supply the driving power.

13 is a state diagram in which a safe mode of the augmented reality system 1 operates.

The augmented reality system 1 can be set to a safe mode for the user's safety.

When the safety mode is set, the augmented reality terminal 100 senses an actual object approaching the user through the front cameras 121 and 123. [ That is, it is possible to display a dangerous situation on the transparent display 110 by sensing that a real object, such as a car, a bicycle, etc., which is dangerous to the user, is approaching quickly to the user's direction.

Referring to FIG. 13, a screen in a state in which a user is looking forward is shown. A rectangular area indicated by a dotted line in the center is defined as a sight line concentrated area, and the area is defined by a dotted line frame. In this case, when the front cameras 121 and 122 detect that the actual object, which may be a danger to the user such as a car or a bicycle, approaches the user in a fast direction (more than a predetermined speed)

The size of the sight line concentrated area is automatically expanded and the virtual object displayed on the screen automatically moves to the outward direction of the sight line concentrated area or the transparency is further strengthened so that the user can easily recognize the actual object accessed by the user do.

The size of the visual attention zone, the transparency of the virtual object, and the moving speed of the virtual object moving in the outward direction can be automatically determined in proportion to the speed of the actual object accessing the user (proportional to the direct proportion or square).

Further, when the recognition camera 145 detects the direction of the user's eyes, the eye-focused area operates to automatically move in accordance with the direction of the eyes. That is, when the eye of the user is gazing to the right, the eye-focused area is moved to the right direction. In this case, when the front cameras 121 and 123 detect that an actual object, which may be a danger to the user such as an automobile or a bicycle, is approaching from the front to the user, the sight line concentrated area may include a safety operation And the eye-focused area automatically moves in the front direction of the user.

That is, the eye-focused area may be set to automatically move in the direction of the actual object that is rapidly approaching the user.

In addition, a new virtual object may be assigned to a real object that may be a risk to the user, and a virtual line may be displayed to indicate the association with a physical object. At this time, a new virtual object may be displayed as an icon, a character indicating danger, and the access speed may be additionally displayed as a virtual object.

For reference, the recognition camera 145 can detect the movement of the user's pupil, the gaze direction of the pupil, and the change in the size of the eyes, and thus can instruct an operation command based on the change in size of the pupil.

For example, every time the user opens his or her eyes for a predetermined time, the virtual information corresponding to the lower information is gradually displayed, and the virtual information corresponding to the upper information is gradually displayed every time the eyes are opened for a predetermined time Can be instructed. Further, in order to improve the recognition rate of the recognition camera 145, an instruction eyebrow may be attached to a user's eyebrow. The pointing eyebrow may be coated with a reflective paint that reflects a predetermined infrared wavelength, and the recognition camera 145 may be configured to recognize the infrared wavelength to improve the command recognition rate.

14 is a state diagram in which the overhead view mode of the augmented reality system 1 operates.

Referring to FIG. 14, an augmented reality system 1 may have an overhead view mode.

The overhead view mode refers to a mode in which a composite image photographed on the user's head is displayed on the transparent display 110.

That is, although not shown in the figure, a plurality of view cameras capable of photographing an infrared region and a visible light region may be additionally arranged along the spectacle frame. Accordingly, the images photographed by the plurality of view cameras can be synthesized and provided to the user's field of view. In addition to the daytime, especially at night, the user can safely move the footprint. At this time, the height of the ground surface based on a predetermined previous position is displayed on the soles of the feet, which can help the user to move more safely.

In addition, the augmented reality system 1 may further include a plurality of sensing units 300, each of which includes a Wi-Fi communication module and is disposed at a predetermined interval in a room.

The plurality of sensing units 300 may be selectively arranged to sense the position of the augmented reality terminal 100 in the room,

The plurality of sensing units 300 transmits the sensing information to the server 200 every time it detects a WIFI HOTSPOT signal periodically output from the augmented reality terminal 100, The relative position of the augmented reality terminal 100 can be determined on the basis of the absolute position of the sensing unit 300.

As described above, in the proposed system, the present position information is acquired based on the satellite position information in the outdoor, and the current position information is acquired using the Wi-Fi signal in the room.

An additional method for acquiring current position information in indoor and outdoor is described as follows.

The method using the Wi-Fi signal can be basically divided into triangulation and fingerprinting.

First, triangulation measures the received signal strength (RSS) from three or more access points (APs), converts the signal intensity to distance, and calculates the position through the equation.

Next, in the fingerprinting method, the indoor space is divided into small cells, and signal intensity values are directly collected from each cell, and a database is constructed to form a radio map. Then, It estimates the cell that shows the most similar signal pattern compared with the database to the user's location.

Next, a method of collecting position data of each smartphone while exchanging Wi-Fi signals directly or indirectly from a plurality of users having smart phones in the vicinity may be used.

In addition, since the communication module 142 of the augmented reality terminal 100 includes the Bluetooth communication module, the current location information can be grasped by using the Bluetooth communication.

First of all, there is a method of estimating that a user is located around a beacon when communicating with any one beacon after arranging a plurality of beacons in an indoor space.

Next, a receiver in which a plurality of directional antennas are arranged on the surface of a hemispherical shape is disposed in an indoor space, and then a position of a user is estimated through identification of a specific directional antenna that receives a signal transmitted from the augmented reality terminal 100 . In this case, when two or more receivers are arranged, the user's position may be identified in a three-dimensional form.

In addition, the current position information of the user is grasped based on the satellite position information, and the velocity and the moving direction are estimated using the information of the 9-axis sensor 143 when entering the shadow area of the satellite position information. That is, the position of the user can be estimated through indoor step counting, stride length estimation, and heading estimation. At this time, in order to increase the accuracy of the estimated information, information such as the user's physical condition (key, weight, stride) may be input and used for the position estimation calculation. When using the information of the 9-axis sensor 143, the accuracy of the estimated information can be improved by combining the position estimation technique using the Wi-Fi communication.

In other words, even if the accuracy is low with the Wi-Fi technique, the absolute position is globally calculated and the relative position of the locally high accuracy is obtained through the information of the 9- Can be combined to improve accuracy. In addition, the accuracy of the position estimation information may be further improved by further applying the Bluetooth communication method.

In addition, since the magnetic field information unique to the position is formed in the indoor space, a method of estimating the current position similarly to the fingerprinting method using Wi-Fi can be applied after constructing the magnetic field map of each space. At this time, the trend of the change in the magnetic field generated when the user moves can be used as additional information.

There is also a method of utilizing the illumination installed in the indoor space. That is, a method of outputting a specific position identifier while blinking the LED illumination at a speed that the person can not identify, and a method of the camera of the augmented reality terminal 100 recognizing the specific position identifier and estimating the position can be used.

In addition, there is a technique to database the images taken from various locations and various angles of the indoor space, and then to match the pictures taken at the user's location. In addition, various landmarks (signs, trademarks, room numbers, signs, etc.) A method of correcting the position can be used.

For reference, it is most preferable that the method of acquiring the current position information in the indoor space improves the accuracy by combining at least one of the above methods.

The augmented reality system according to the embodiment of the present invention can correctly identify the existence or nonexistence of the entity object in the photographed image and display the virtual object allocated to the actual object at the target position.

That is, it is possible to display the position of the virtual object at the correct position by simultaneously applying the space recognition and the object recognition.

In addition, the augmented reality system recognizes the coordinates of real objects while generating space-based 3-dimensional matching coordinate system based on spatial recognition, and simultaneously generates 3-dimensional matching coordinate system based on object recognition based on object recognition, The error can be corrected by correcting the matching coordinate system. Also, it is possible to more precisely grasp the position of the actual object in the room or outdoor by applying the technique of distinguishing the real object based on the current position of the user. Also, the augmented reality system can be constructed by applying at least one technique for identifying similar objects, a technique for maintaining distance between objects, and a technique for adjusting the amount of information of virtual objects according to distance.

In addition, the augmented reality system 1 according to the embodiment of the present invention can dynamically adjust the information amount of a virtual object according to the distance between the object and the user, so that the user can conveniently check information of the desired virtual object.

Therefore, when a virtual object is displayed in the form of an augmented reality advertisement, the time for the user to recognize the virtual object is concentrated, and the advertisement effect can be increased.

Also, the augmented reality system according to the embodiment of the present invention can select an additional recognition area and identify similar objects by assigning unique identifiers to real objects based on image differences in the additional recognition area.

Also, the augmented reality system can identify each real object by considering both the unique identifier assigned to each real object and the current position information of each real object based on the image difference of the additional recognition area.

Therefore, even if real objects having high similarity are arranged, it is possible to identify them and display the virtual objects allocated to the respective real objects, thereby conveying information without confusion to the user.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: augmented reality terminal
200: Server
300: sensing part
110: Display
121: Left front camera
122: Right front camera
131: Left 3D sensor
132: Right 3D sensor
141: Satellite module
142: Communication module
143: 9 axis sensor
144: Battery
145: Recognition camera
150:

Claims (6)

delete delete A three-dimensional virtual image corresponding to the current position information and the current position information is displayed on a display area of the user, An augmented reality terminal for displaying the augmented reality terminal; And a server for real-time providing the three-dimensional virtual image corresponding to the actual image information to the augmented reality terminal in real time based on the current position information transmitted from the augmented reality terminal, The augmented reality terminal recognizes a space three-dimensionally and provides space-based three-dimensional virtual images of three-dimensional virtual images assigned to the identified real objects, And a display unit for displaying each virtual object on the coordinate of the real object allocated in advance based on the space-based three-dimensional matching coordinate system, wherein each time the actual object of the real image information is image- The coordinates of the actual object are grasped again, and the coordinates of the actual object are grasped, and the coordinates of the updated actual object Displays virtual objects pre-assigned to coordinates,
The server selects an additional recognition area by subtracting each of the actual image information from real objects having a predetermined image similarity value or more among a plurality of actual objects existing in the actual image information, Identifies each real object by giving a unique identifier to each of the actual objects, taking into consideration both the unique identifier given to each actual object and the current position information of each actual object based on the image difference of the additional recognition area, Augmented reality system.
A three-dimensional virtual image corresponding to the current position information and the current position information is displayed on a display area of the user, An augmented reality terminal for displaying the augmented reality terminal; And a server for real-time providing the three-dimensional virtual image corresponding to the actual image information to the augmented reality terminal in real time based on the current position information transmitted from the augmented reality terminal, The augmented reality terminal recognizes a space three-dimensionally and provides space-based three-dimensional virtual images of three-dimensional virtual images assigned to the identified real objects, And a display unit for displaying each virtual object on the coordinate of the real object allocated in advance based on the space-based three-dimensional matching coordinate system, wherein each time the actual object of the real image information is image- Dimensional coordinate system based on the object recognition based on the coordinate of the actual object, A reference three-dimensional coordinate system, and matching the updated three-dimensional coordinate system of the space-based matching,
The server selects an additional recognition area by subtracting each of the actual image information from real objects having a predetermined image similarity value or more among a plurality of actual objects existing in the actual image information, A unique identifier is assigned to each of the real objects, and a candidate position of the additional recognition area is determined based on the current position information of each real object.
A three-dimensional virtual image corresponding to the current position information and the current position information is displayed on a display area of the user, An augmented reality terminal for displaying the augmented reality terminal; And a server for real-time providing the three-dimensional virtual image corresponding to the actual image information to the augmented reality terminal in real time based on the current position information transmitted from the augmented reality terminal, The augmented reality terminal recognizes a space three-dimensionally and provides space-based three-dimensional virtual images of three-dimensional virtual images assigned to the identified real objects, And a display unit for displaying each virtual object on the coordinate of the real object allocated in advance based on the space-based three-dimensional matching coordinate system, wherein each time the actual object of the real image information is image- Dimensional coordinate system based on the object recognition based on the coordinate of the actual object, A reference three-dimensional coordinate system, and matching the updated three-dimensional coordinate system of the space-based matching,
The server selects an additional recognition area by subtracting each of the actual image information from real objects having a predetermined image similarity value or more among a plurality of actual objects existing in the actual image information, Identifies each real object by giving a unique identifier to each of the actual objects, taking into consideration both the unique identifier given to each actual object and the current position information of each actual object based on the image difference of the additional recognition area, Augmented reality system.
A three-dimensional virtual image corresponding to the current position information and the current position information is displayed on a display area of the user, An augmented reality terminal for displaying the augmented reality terminal; And a server for real-time providing the three-dimensional virtual image corresponding to the actual image information to the augmented reality terminal in real time based on the current position information transmitted from the augmented reality terminal, The augmented reality terminal recognizes a space three-dimensionally and provides space-based three-dimensional virtual images of three-dimensional virtual images assigned to the identified real objects, And a display unit for displaying each virtual object on the coordinate of the real object allocated in advance based on the space-based three-dimensional matching coordinate system, wherein each time the actual object of the real image information is image- The coordinates of the actual object are grasped again, and the coordinates of the actual object are updated, and the coordinates of the updated actual object Displays virtual objects pre-assigned to coordinates,
The server selects an additional recognition area by subtracting each of the actual image information from real objects having a predetermined image similarity value or more among a plurality of actual objects existing in the actual image information, A unique identifier is assigned to each of the real objects, and a candidate position of the additional recognition area is determined based on the current position information of each real object.

Images