KR20190081034A - An augmented reality system capable of manipulating an augmented reality object using three-dimensional attitude information and recognizes handwriting of character - Google Patents

Abstract

According to the present invention, an augmented reality system capable of recognizing the writing of characters and manipulating an augmented reality object includes a mobile terminal. For displaying a three dimensional virtual image on the display, the mobile terminal displays a virtual object corresponding to the content of characters after recognizing the characters when a dotted line guide is displayed on the edge of the characters displayed on a display and writing is sensed along the dotted line guide; and when the virtual object is touched, reproduces the motion of a preset virtual object corresponding to the touched part.

Description

[0001] The present invention relates to an augmented reality system capable of recognizing the handwriting of a character and manipulating the augmented reality object,

The present invention relates to an augmented reality system, and more particularly, to an augmented reality system capable of recognizing handwriting of a character and manipulating an augmented reality object.

Recently, contents are being actively researched using augmented reality technique in which various information is superimposed on a real space when photographing using a camera module.

Augmented Reality (AR) is a technology belonging to one of the fields of Virtual Reality (VR). It combines a virtual environment with a real environment that a user feels in a sense, and feels as if a virtual environment exists in the original real environment Is a computer technique.

This augmented reality is advantageous in that it can supplement and supplement additional information that is difficult to obtain only in the real world by synthesizing virtual objects on the basis of the real world, unlike the existing virtual reality which only targets virtual space and objects.

That is, a realization realized by matching a virtual environment created by computer graphics to a real image viewed by a user, for example, a three-dimensional virtual environment, can be called an augmented reality. Here, the 3D virtual environment provides necessary information from the real image that the user views, and the 3D virtual image matches with the real image to increase the user's immersion.

Such augmented reality can provide better realism by making the distinction between the real environment and the virtual environment be blurred by providing the real image in the three-dimensional virtual environment as well as the simple virtual reality technology.

Due to the advantages of such an augmented reality system, active research and development are currently under way for technologies that have been combined with augmented reality around the world. For example, augmented reality technology is being put into practical use in fields such as broadcast, advertisement, exhibition, game, theme park, military, education and promotion.

Recently, the handheld augmented reality system has become possible by improving the computing ability of mobile devices such as mobile phones, personal digital assistants (PDAs), and ultra mobile personal computers (UMPCs) and wireless network devices.

As these systems become available, many augmented reality applications using mobile devices have been developed. In addition, the spread of mobile devices is becoming very common and an environment for accessing augmented reality applications is being created.

In addition, user demands for various supplementary services using an augmented reality of a terminal are increasing, and attempts to apply various augmented reality contents to a terminal user are increasing.

Korean Patent Laid-Open Publication No. 10-2016-0092292 relates to a system and method for providing an augmented reality service for an advertisement, and it is an object of the present invention to apply an augmented reality to an advertisement, And the system is able to communicate effectively by being intrigued and intriguing.

Meanwhile, various wearable device products that can be worn on the body of the user are being released. In particular, a system has been developed in which mobile technology and transparent display technology are applied to glasses to allow users to check various information through glasses.

1 is a diagram illustrating a virtual object displayed in a conventional augmented reality system.

Referring to FIG. 1, the conventional augmented reality system displays too many virtual objects in a transparent display of eyeglasses, thereby hindering the user's eyesight. For this reason, it is difficult for the user to recognize necessary information.

In addition, if there are a plurality of actual objects having similar appearance, duplicate recognition or unassigned virtual objects may be displayed to cause confusion, so that a technology capable of identifying duplicate objects and preventing duplicate recognition is required.

In addition, when augmented reality is driven by a mobile terminal to move / rotate a virtual object, a virtual object must be operated by touching a touch screen of a smart phone with a finger.

Since the touch screen (display) provides only two-dimensional coordinates of the finger touch point, the virtual object can be moved horizontally or vertically on a two-dimensional plane. In the case of the rotating operation, the virtual object can be rotated with respect to the one-dimensional axis through the finger gesture.

Therefore, in the conventional art, it is difficult to efficiently manipulate a virtual object three-dimensionally because only a limited one-dimensional or two-dimensional manipulation is possible. In order to change the virtual object to various poses, it is necessary to change the movement or rotation mode frequently, so that it takes much time and requires a lot of fingers.

On the other hand, techniques for recognizing characters include online character recognition, which is a method of recognizing characters during writing by a user, and offline character recognition, which is a method of recognizing already created handwritten characters.

Since online character recognition can obtain dynamic information of time and space of handwriting, information such as stroke number, stroke order, and handwriting direction can be known. On the other hand, offline character recognition can not obtain dynamic information, and recognizes characters through processes such as binarization, contour extraction, and shape analysis or template matching.

In the enhancement technology based on character recognition, offline character recognition is mainly used. However, due to the characteristics of offline character recognition, there is a problem that the recognition rate is significantly lowered if the written character differs from the learning character shape. Especially for children, the character recognition rate is very different because it is very different from the standard writing method. Therefore, most of the character learning augmented reality applications are using character cards.

KR 10-2016-0092292 A

SUMMARY OF THE INVENTION The present invention has been proposed in order to solve the above-described technical problems, and it is an object of the present invention to provide a game machine, Thereby providing an augmented reality system for displaying an object.

According to an embodiment of the present invention, when displaying a three-dimensional virtual image on a display, a dotted line guide is displayed on a border of a character displayed on the display, and a handwriting is detected along the dotted line guide And a mobile terminal for recognizing the character and displaying a virtual object corresponding to the character content. When the virtual object is touched, the motion of the preset virtual object corresponding to the touched portion is reproduced An augmented reality system is provided.

Also, the mobile terminal of the present invention includes a 9-axis sensor to acquire its own three-dimensional attitude information, and the attitude of at least one or more virtual objects selected in the mobile terminal is changed in synchronization with the three- .

According to another embodiment of the present invention, there is provided a video camera for capturing an image of a user and acquiring real image information, and displaying a three-dimensional virtual image on a display, And displaying the 3D virtual image on the display, wherein the mobile terminal displays a character corresponding to a name of the 3D virtual image on the display, wherein a dotted line guide is displayed on a border of the character And when a handwriting is sensed along the dotted line guide, recognizing the character, a virtual object corresponding to the character content is displayed, and when the virtual object is touched, movement of a preset virtual object corresponding to a touched portion Is reproduced in the augmented reality system.

The mobile terminal includes a 9-axis sensor to acquire three-dimensional attitude information of the mobile terminal, and the attitude of at least one or more virtual objects selected in the mobile terminal is changed in synchronization with the three-dimensional attitude information. do.

According to another embodiment of the present invention, there is provided an image display apparatus including a video camera for capturing an image of a user and acquiring real image information, and displaying a three-dimensional virtual image on a display, A mobile terminal for displaying the corresponding 3D virtual image on the display; And a server for providing the mobile terminal with the current location information and the 3D virtual image corresponding to the actual image information transmitted from the mobile terminal in real time, Wherein the mobile terminal displays a character corresponding to the name of the three-dimensional virtual image on the display, wherein the character corresponding to the name of the three-dimensional virtual image is displayed on the display, When a handwriting is sensed along the dotted line guide, a virtual object corresponding to the character content is displayed after recognizing the character, and when the virtual object is touched, The motion of the corresponding preset virtual object is reproduced, and the mobile terminal transmits the 9-axis sensor Dimensional attitude information of the mobile terminal, and the attitude of at least one or more virtual objects selected in the mobile terminal is changed in synchronization with the three-dimensional attitude information.

The coordinates of the at least one virtual object selected in the mobile terminal included in the present invention are changed from the spatial coordinate system for the actual image information to the mobile coordinate system of the mobile terminal, And the spatial coordinate system for the actual image information is changed in the mobile coordinate system of the terminal.

In the augmented reality system according to the embodiment of the present invention, when a handwriting is detected along a dotted line guide, a dotted line guide is displayed on a border of a character displayed on a display, and a virtual object corresponding to the character content is displayed .

That is, the user can perform handwriting practice by providing a dotted line guide, and the character font of the dotted line guide is standardized, so that the character recognition rate is high. In particular, children between 0 and 5 years old can learn a specific character regardless of the character, because they recognize it as an object, that is, a picture or a lump, instead of recognizing a character as an adult. Writing letters can be done by drawing or coloring. In addition, learning can be done using multidisciplinary information in a three-dimensional manner. Children express, develop and express their senses in a way that they see, hear, and feel. In the present invention, it is possible to increase the perception ability through the expression method depending on the senses, thereby enabling sensory immersion for information.

Here, the sensory immersion refers to how much the user's attention falls into the information of the virtual world before the eyes. For example, when the object is augmented in augmented reality, the children concentrate and observe the complete image of the object Touch the screen with your finger or move the camera. By providing a real-world interface that allows children to manipulate virtual objects through specific drawing (writing activities) activities, it can induce natural behavior in learning situations and immerse themselves in learning itself.

In addition, since the attitude of at least one or more virtual objects selected in the augmented reality system according to the embodiment of the present invention is changed in synchronization with the three-dimensional attitude information of the mobile terminal, the attitude of the virtual object can be changed variously with a minimum of actions.

In addition, the selected virtual object can be moved / rotated three-dimensionally using the three-dimensional movement / rotation information of the mobile terminal. That is, the virtual object can be easily moved / rotated by changing the attitude of the mobile terminal without touching the touch screen (display) many times without converting the moving and rotating operation modes.

In addition, it is possible to record the motion of the virtual object, to store and reproduce the movement of the virtual object, and to improve the utilization such as the augmented reality animation implementation.

1 is a diagram showing a virtual object displayed in a conventional augmented reality system;
FIG. 2 is a configuration diagram of an augmented reality system 1 according to an embodiment of the present invention, and FIG. 2A is a flowchart showing an operation process of the augmented reality system 1. FIG.
FIG. 3 is a diagram showing an example in which a dotted line guide is shown at the edge of a character
FIGS. 4 and 5 illustrate an example in which a virtual object corresponding to text content is displayed after recognizing a character
6 and 6A are diagrams illustrating an example of moving a virtual object in the AR system 1
7 is a diagram showing another example of moving a virtual object in the augmented reality system 1
8 is a diagram showing a condition for selecting a virtual object in the AR system 1
9 is a flowchart showing a learning process for identifying similar objects in the augmented reality system 1. Fig.
9A is a diagram showing a process of selecting an additional recognition area for identifying a similar object in the AR system 1
10 is a flowchart showing a process of identifying similar objects in the augmented reality system 1
11 is a first example showing a state for identifying a similar object in the augmented reality system 1. Fig.
12 is a second exemplary diagram showing a state for identifying a similar object in the augmented reality system 1. Fig.
13 is a diagram showing another operation principle of the augmented reality system 1
Figs. 14 and 14A are diagrams showing another operation principle of the augmented reality system 1
15 is a configuration diagram of the mobile terminal 100 of the augmented reality system 1. Fig.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, in order to facilitate a person skilled in the art to easily carry out the technical idea of the present invention.

FIG. 2 is a block diagram of an augmented reality system 1 according to an embodiment of the present invention, and FIG. 2A is a flowchart illustrating an operation process of an augmented reality system 1. Referring to FIG.

The augmented reality system 1 according to the present embodiment includes only a brief configuration for clearly explaining the technical idea to be proposed.

2 and 2A, the augmented reality system 1 includes a mobile terminal 100, a server 200, and a plurality of sensing units 300. Here, the server 200 and the plurality of sensing units 300 may be selectively provided in the augmented reality system 1 according to the embodiment.

The augmented reality system 1 according to the embodiment of the present invention can recognize the handwriting of a character and operate to enable manipulation of an augmented reality object.

That is, when displaying the three-dimensional virtual image on the display, the mobile terminal 100 displays a dotted line guide on the edge of the character displayed on the display. When the handwriting is detected along the dotted line guide, the mobile terminal 100 recognizes the character, Displays the corresponding virtual object.

At this time, when the virtual object is touched, the movement of the preset virtual object corresponding to the touched portion can be reproduced. Also, since the mobile terminal 100 includes the nine-axis sensor, it acquires its own three-dimensional attitude information, and the attitude of at least one or more virtual objects selected from the mobile terminal 100 can be changed in synchronization with the three- .

That is, when the user writes along the dotted line guide, the character is recognized and the corresponding content is augmented and displayed as a virtual object. The content to be augmented may include an image, a moving image, an animated 3D model, and a voice. In the case of reactive content, when the user performs an action such as touching, the content operates to express animation, voice, sound,

Meanwhile, in the mobile terminal 100, an image camera for capturing an image of a user around the user and acquiring actual image information is built therein. In displaying the three-dimensional virtual image on the display, the mobile terminal 100 displays three- Display the image on the display.

The server 200 provides the mobile terminal 100 with real time 3D virtual images corresponding to the current location information and actual image information transmitted from the mobile terminal 100. [

At this time, the mobile terminal 100 can display the character corresponding to the name of the three-dimensional virtual image on the display without directly displaying the three-dimensional virtual image provided from the server 200. [

At this time, the dotted line guide is displayed on the edge of the character, and when the handwriting of the user is detected along the dotted line guide, the character is recognized and then the virtual object corresponding to the character content is displayed. At this time, when the virtual object is touched, the movement of the preset virtual object corresponding to the touched portion can be reproduced.

The mobile terminal 100 is basically configured to provide satellite position information to the server 200 as current position information. When the communication module is included in the mobile terminal 100, not only the satellite position information, but also the location of the nearby Wi-Fi repeater, the location of the base station, and the like can be additionally provided to the server 200 as the current location information .

For example, since the satellite position information can not be received indoors in particular, the mobile terminal 100 may further grasp the signal strength of at least one of the searched Wi-Fi repeaters and send it to the server 200 . That is, since the absolute position of the Wi-Fi repeater located indoors is previously stored in the server 200, the mobile terminal 100 may additionally provide the unique number and signal strength of the found Wi-Fi repeater , The server 200 can grasp the relative movement path of the mobile terminal 100.

That is, the relative distance between the mobile terminal 100 and the Wi-Fi repeater can be confirmed by the signal strength, and the moving direction can be calculated based on the change in the signal strength with the neighboring Wi-Fi repeater. An additional method for acquiring the current position information in the room will be described later.

Therefore, the server 200 recognizes the virtual objects allocated to the respective real objects through the current location information of the user having the mobile terminal 100 and the actual image information captured by the video camera of the mobile terminal 100, And transmits the information to the mobile terminal 100 in real time.

The server 200 identifies the actual objects of the actual image information and provides the virtual objects of the three-dimensional virtual images assigned to the identified actual objects to the mobile terminal 100, And attitude of at least one or more virtual objects selected in the mobile terminal 100 can be changed in synchronization with the three-dimensional attitude information.

At this time, the coordinates of at least one or more virtual objects selected from the mobile terminal 100 are changed from the spatial coordinate system for the real image information to the mobile coordinate system of the mobile terminal 100, and the coordinates of the de- ) To the spatial coordinate system of the actual image information in the mobile coordinate system. A detailed description thereof will be given later.

The augmented reality system 1 is configured to perform the role of the server 200 in the mobile terminal 100 itself without the configuration of the server 200 when the computing ability and the storage space of the mobile terminal 100 are sufficient . That is, the mobile terminal 100 has a built-in video camera that captures an image of a user around the user and acquires actual image information, and displays a three-dimensional virtual image on the display. The three- Display the image on the display. At this time, the mobile terminal 100 may be provided with a nine-axis sensor to obtain its three-dimensional attitude information. Therefore, the attitude of at least one or more virtual objects selected from the mobile terminal 100 is changed in synchronization with the three-dimensional attitude information.

When the user moves the mobile terminal 100 three-dimensionally, the selected virtual object is automatically changed in posture in synchronization with the three-dimensional movement. For reference, a 9-axis sensor is referred to as a 9-axis sensor because a value of 9 axes is measured as three axes of acceleration, 3 axes of inertia and 3 axes of geomagnetism, and a temperature sensor may be additionally provided for correcting the temperature value . The 9-axis sensor not only senses the three-dimensional movement of the mobile terminal but also can sense the gaze direction, movement direction, tilt, etc. of the user.

Fig. 3 is a diagram showing an example in which a dotted line guide is displayed at the edge of a character.

3, "elephant", "pig", "wolf", and "eagle" are displayed in English on the display of the mobile terminal 100, and a dotted line guide is displayed on the rim of each character.

Accordingly, when the user performs a touch operation or draws with the pen along the dotted line guide, that is, when the handwriting is detected along the dotted line guide, the mobile terminal 100 recognizes the character, Display.

At this time, the type of the language can be displayed so that various choices can be made. In this embodiment, English and Korean are selectively displayed, and characters are displayed in the corresponding language. However, various languages including English and Korean may be selected according to the embodiment.

FIGS. 4 and 5 illustrate examples in which a virtual object corresponding to text content is displayed after recognizing a character.

4 and 5, when a handwriting of a character is detected for a zebra and a wolf, a virtual object corresponding to the character content, that is, a zebra or a wolf is displayed as a virtual object do.

At this time, when the virtual object is touched, the movement of the preset virtual object corresponding to the touched portion can be reproduced. For example, if you touch the zebra's butt, the action of the zebra can be played, and when the zebra's head is stroked, the action of rocking the head up and down can be played.

Further, when the character "love" is additionally handwritten while the zebra is displayed as a virtual object, the meaning of "love" is given to the action of the zebra, and the action for the "love" of the zebra set in advance can be reproduced. At this time, the dotted line guide is also displayed on the border of the character "love".

Also, when touching the zebra's hips with the zebra being displayed as a virtual object, the character "no" or "surprise" may be displayed on the display while the action of the zebra is played back.

Meanwhile, the mobile terminal 100 may be provided with an additional camera capable of photographing the user's face in the display direction. This camera can capture three-dimensional images, so that the user's body, especially three-dimensional images (depth images) of the face can be grasped.

Therefore, when the body interlocking mode is set, the facial expression of the user can be detected. If the smiling expression of the user is detected while the zebra is displayed as a virtual object, an action for the "joy" have. If the user's crying expression is detected, an action for the "grief" of the preset zebra can be reproduced. If an angry facial expression of the user is detected, the action for the "anger" of the preset zebra will be reproduced.

The augmented reality system 1 capable of recognizing the handwriting of the character and manipulating the augmented reality object can learn using the three-dimensional multi-sensory information. Children express, develop and express their senses in a way that they see, hear, and feel. In the present invention, it is possible to increase the perception ability through the expression method depending on the senses, thereby enabling sensory immersion for information.

6 and 6A are views illustrating an example of moving a virtual object in the augmented reality system 1. In FIG.

6 and 6A, in a first step (STEP 1), virtual objects are augmented on the space based on the spatial coordinate system and displayed on the display of the mobile terminal 100. [

If the user touches one or more virtual objects displayed on the display (touch screen) of the mobile terminal 100 in the second step (STEP 2), the selected virtual object is selected by the user And is connected to the mobile terminal 100. Here, the meaning of the connection means that the reference coordinate system of the virtual object is changed from the spatial coordinate system to the coordinate system of the mobile terminal 100.

When the mobile terminal 100 is moved / rotated in a state where the virtual object is touched in the third step (STEP 3), movement / rotation information of the virtual object connected to the mobile terminal 100 is transmitted to the mobile terminal 100 in three- In a synchronous manner.

If the virtual object is not touch input in the fourth step (STEP 4), the virtual object is separated from the mobile terminal 100. Here, the separation means that the reference coordinate system of the virtual object is changed from the coordinate system of the mobile terminal 100 to the spatial coordinate system.

In the fifth step (STEP 5), the posture change image of the virtual object changed in synchronization with the three-dimensional posture information of the mobile terminal 100 is stored and can be reproduced later. That is, if the posture change image of the virtual object is stored, the recorded information can be retrieved and played back by pressing the play button in the future.

Only the reference coordinate system of the touched virtual object is stored in the spatial coordinate system of the mobile terminal 100 in the second step (STEP 2) to the fourth step (STEP 4) of the above- The description has been given of an example in which the coordinate system is changed to cooperate with the three-dimensional motion of the mobile terminal 100,

According to an embodiment, when a virtual object is touched for a predetermined time or longer, a virtual object is selected, and when the user touches again for a predetermined time or longer, the selection of the virtual object is released.

FIG. 7 is a diagram illustrating another example of moving a virtual object in the AR system 1. FIG.

Referring to FIG. 7, when augmented content is loaded in the mobile terminal 100 and a virtual object is displayed, the user selects a virtual object and changes the coordinate system of the virtual object to the mobile coordinate system. Thereafter, the selected virtual object moves in conjunction with the three-dimensional movement of the mobile terminal 100.

8 is a diagram showing a condition for selecting a virtual object in the augmented reality system 1. Fig.

Referring to FIG. 8, when selecting a virtual object displayed on the display of the mobile terminal 100, a virtual object is selected when a virtual object is touched for a predetermined time t0 or more. When the virtual object is touched again for a predetermined time t0, The way in which the object is deselected may be applied.

At this time, it is preferable that the display is configured to be recognized as a touch valid period of the virtual object only when the display is touched with a predetermined pressure (k1) or more.

That is, when a virtual object is touched for a predetermined time t0 or more, a virtual object is selected, the reference coordinate system of the touched virtual object is changed from the spatial coordinate system to the coordinate system of the mobile terminal 100, .

At this time, the time at which the posture change image of the virtual object is stored may be automatically set according to the duration after touching the predetermined time t0 or longer.

8, if the touch is continued for the first time t1 even after the virtual object is touched for the predetermined time t0, the interval P1 from the predetermined time t0 to the first time t1 ), The storage time is automatically set. Therefore, as the touch time of the virtual object becomes longer, the storage time becomes longer and the storage time is set as a multiple of the touch time. At this time, the storage time is counted from the initial movement time of the virtual object.

At this time, the automatically set storage time is displayed in the form of time bar on the display, and the required time is displayed in real time. At this time, when the user moves left and right by dragging the time bar, the time required is increased or shortened. That is, the storage time automatically set by the drag operation of the user can be increased or decreased.

Meanwhile, the mobile terminal 100 may be provided with an additional camera capable of photographing the user's face in the display direction. This camera can capture three-dimensional images, so that the user's body, especially three-dimensional images (depth images) of the face can be grasped.

Therefore, when the body interlocking mode is set, the camera can grasp at least one of the head movement of the user, the finger movement, and the movement of the pupil three-dimensionally, and the virtual object can be set to move in synchronization with the movement . At this time, the head movement, the finger movement, and the pupil movement can be individually selected, and a plurality of the head movement, the finger movement, and the pupil movement can be selected to grasp the movement.

Meanwhile, the augmented reality system 1 according to the embodiment of the present invention can identify the similar object by selecting an additional recognition area and assigning a unique identifier to the actual object based on the image difference of the additional recognition area.

In addition, the augmented reality system 1 can identify each real object by considering both the unique identifier given to each real object and the current position information of each real object based on the image difference in the additional recognition area.

Therefore, even if real objects having high similarity are arranged, it is possible to identify them and display the virtual objects allocated to the respective real objects, thereby conveying information without confusion to the user.

The visual cognition description of an object can be divided into four steps. - DCRI (Detection, Classification, Recognition, and Identification) -

First, detection is a step that can detect whether or not an object exists.

Next, Classification is a step of knowing what classification an object is. - For example, it can be classified as human or animal -

Next, Recognition is a step to know the approximate characteristics of the object. - For example, you can find brief information about clothes that a person wears -

Finally, Identification is a step that allows to know the detailed characteristics of an object. For example, you can identify a person's face and know the number of the license plate.

The augmented reality system 1 of the present invention operates so that an identification step can be implemented to distinguish detailed characteristics of similar real objects.

For example, the augmented reality system 1 recognizes a character attached to a facility (real object) having a similar shape and assigns a unique identification number, or identifies a difference portion, and then selects the additional recognition region The 2D / 3D feature information difference of the additional recognition area, and the current position information measured by the satellite position information and the Wi-Fi signal, to distinguish real objects having high similarity values.

FIG. 9 is a flowchart illustrating a learning process for identifying similar objects in the AR system 1. FIG. 9A illustrates a process for selecting an additional recognition area for identifying similar objects in the AR system 1. FIG. .

9 and 9A, the server 200 operates to identify a real object of actual image information and to provide a virtual object of a three-dimensional virtual image, each of which is assigned to an identified real object, to the mobile terminal 100. [

That is, the server 200 selects the additional recognition area by subtracting the actual image information from the actual objects having the predetermined image similarity value d or more among the plurality of actual objects existing in the actual image information, And assigns a unique identifier to the actual object based on the image difference of the object.

For example, when characters or numbers different from each other are displayed in the additional recognition area, the server 200 assigns a unique identifier to each of the actual objects based on the difference of the additional recognition areas, stores the database in a database, And may transmit the assigned virtual object to the additional mobile terminal 100.

That is, when a plurality of actual objects have a predetermined image similarity value (d) or more, the image is abstracted and the image is differentiated to set an additional recognition area (additional learning area) And each unique identifier is assigned.

FIG. 10 is a flowchart illustrating a process of identifying similar objects in the AR system 1, and FIG. 11 is a first exemplary view illustrating a state in which similar objects are identified in the AR system 1. FIG.

Referring to FIGS. 10 and 11, when actual image information is transmitted from the mobile terminal 100, the server 200 distinguishes real objects of actual image information, and then, when there is no similar image The virtual object corresponding to the identified image is allocated to each of the virtual objects.

At this time, the server 200 compares the information of the additional recognition area with the similar image (if a similar object exists), identifies the unique identifier, and then assigns the virtual object corresponding to the unique identifier.

11, when a facility (a real object) having a similar shape is disposed at a neighboring location, the server 200 recognizes a plurality of actual objects in the actual image information transmitted from the mobile terminal 100 Then, the information of the additional recognition area of each real object is compared to identify the unique identifier, and the virtual object corresponding to the unique identifier is allocated.

On the other hand, when different identification markers are printed in the additional recognition area of each facility, the shape of the identification markers can be configured as follows.

The identification marker may be configured to include a first identification marker region, a second identification marker region, a third identification marker region, and a fourth identification marker region.

That is, the identification marker recognizes the first identification marker region, the second identification marker region, the third identification marker region, and the fourth identification marker region as one identifier. That is, the mobile terminal 100 basically captures all of the first to fourth identification markers and transmits them to the server 200, and the server 200 recognizes the recognized identification markers as one unique identifier.

At this time, the first identification marker region is configured to reflect the wavelength of the visible light region. That is, the first identification marker area is printed with a common paint and printed so that a human can visually distinguish it.

In addition, the second identification marker region reflects the first infrared wavelength, printed with a paint reflecting the first infrared wavelength, and can not be visually distinguished by a person.

Further, the third identification marker region reflects a second infrared wavelength of a wavelength longer than the first infrared wavelength, and is printed with a paint reflecting the second infrared wavelength, and can not be visually distinguished by a person.

Further, the fourth identification marker region simultaneously reflects the first infrared wavelength and the second infrared wavelength, which is printed with a paint reflecting the first and second infrared wavelengths, and can not be visually distinguished by a person.

At this time, the camera of the mobile terminal 100 for photographing the identification marker is configured to be able to photograph and recognize an infrared wavelength region by mounting a spectral filter for controlling the infrared transmission wavelength.

Therefore, only the first identification marker area printed on the facility can be visually confirmed by the person, and the second identification marker area, the third identification marker area and the fourth identification marker area can not be visually recognized by a person, (Not shown).

Relative printing positions (left, right, top and bottom) of the first identification marker region, the second identification marker region, the third identification marker region and the fourth identification marker region can also be used as one delimiter. In the identification marker area, various characters such as numbers, symbols, and symbols may be printed. In addition, the identification marker may be printed in the form of a QR code or a bar code.

12 is a second exemplary view showing a state in which similar objects are identified in the augmented reality system 1. [

Referring to FIG. 12, the server 200 can identify each real object by considering both the unique identifier given to each real object and the current position information of each real object based on the image difference in the additional recognition area . That is, the actual object can be identified by further considering the current location information of the user (mobile terminal 100).

For example, when a plurality of actual objects maintain a predetermined distance, the user can identify the object using the current location information even if the actual object has a high similarity value. Herein, it is assumed that the current position information includes the absolute position information of the user, relative position information, the moving direction, the acceleration, and the direction of the line of sight.

At this time, the server 200 may select an additional recognition area to further identify actual objects not identified by the current location information, and recognize the difference to identify differences between actual objects.

Also, the server 200 can determine the candidate position of the additional recognition area based on the current position information of each actual object.

That is, referring to FIG. 12, when the user looks at an actual object located at the first position Pl and located at the front,

The server 200 determines the distance between the real object and the user based on the actual image information, and then detects the coordinates (x1, y1, z1) on the space of the actual object.

Since the plurality of additional recognition areas are previously set in the actual object located at the coordinates (x1, y1, z1) on the space, the server 200 stores the coordinates (x1, y1, z1) The candidate position of the additional recognition area can be determined based on the current position information.

The server 200 grasps in advance which real object exists in the coordinates (x1, y1, z1) on the space and which part of the actual object is designated as the additional recognition area, It is possible to identify an object by identifying only a candidate position. Such a scheme has an advantage that the amount of calculation for identifying an additional recognition area can be reduced.

For reference, assuming indoor space and assuming that all directions of illumination are constant, the position and size of shadows due to the illumination may be different from each other even for similar real objects. Accordingly, the server 200 may identify each actual object using the difference between the shadow position and the size of each actual object as additional information based on the current position of the user.

On the other hand, the position of the virtual object of the three-dimensional virtual image, which is allocated and displayed on the actual object of the actual image information, is automatically adjusted to be maintained at a predetermined distance from the actual object and displayed on the mobile terminal 100.

In addition, the position between each virtual object is automatically adjusted and displayed on the mobile terminal 100 so as to maintain a predetermined separation distance.

Therefore, since the position is automatically adjusted so that the objects are not overlapped with each other considering the mutual positional relationship between the actual object and the virtual object, the user can conveniently check the information of the desired virtual object. That is, it takes a long time to recognize the virtual objects with concentration, and the advertisement effect can be increased.

13 is a view showing another operation principle of the augmented reality system 1. Fig.

The operation principle of the augmented reality system 1 will be described in more detail with reference to FIG.

The distance D2 between the first virtual object (Virtual Object 1) and the actual object (Physical Object)

The distance R1 between the center point of the first virtual object 1 and the outermost area of the first virtual object 1 and the distance R1 between the center point of the actual object and the center of the physical object Is set to be longer than the sum of the distance R3 between the outer areas.

The distance Dl between the second virtual object 2 and the physical object is a distance between the center point of the second virtual object 2 and the center of the second virtual object 2 The distance R2 between the outer regions and the distance R3 between the center point of the physical object and the outermost region of the physical object.

The distance D1 between the first virtual object 1 and the second virtual object 2 is calculated by dividing the distance between the center point of the first virtual object 1 and the first virtual object 1 And a distance R2 between the center point of the second virtual object 2 and the outermost region of the second virtual object 2 (Virtual Object 2), which is longer than the sum of the distance R1 between the outermost region of the second virtual object .

The first virtual object (Virtual Object 1) and the second virtual object (Virtual Object 2) move in the horizontal direction and the vertical direction and are automatically overlapped with other objects to automatically adjust the position in the three-dimensional space so as not to deviate from the user's view. , the position is adjusted with respect to the y and z axes.

On the other hand, between the physical object and the first virtual object 1, and between the physical object and the second virtual object 2, the virtual lines L1 and L2 are dynamic As shown in FIG.

When a large number of virtual objects exist on the screen, the virtual lines L1 and L2 are displayed to indicate the association with the physical object, and the thickness, transparency, and color of the virtual lines L1 and L2 are It can be changed automatically according to the line of sight.

For example, when the user gazes at the first virtual object (Virtual Object 1) for a predetermined time or more, the mobile terminal 100 detects whether the first virtual object (Virtual Object 1) The thickness of the virtual line L1 between the first virtual object 1 and the first virtual object 1 is changed to be thicker than the other virtual line L2 and the transparency is also changed to be lower, As shown in FIG.

At this time, assuming that the first virtual object 1 and the second virtual object 2 are all allocated to the physical object, the distance of the plurality of virtual objects allocated to the physical object is As described above, virtual objects having more detailed information are held closer to a physical object while maintaining a predetermined distance D2 and D3, respectively.

For example, if the information of the first virtual object (Virtual Object 1) is more detailed information and the information of the second virtual object (Virtual Object 2) is relatively conceptual information,

The distance D3 between the second virtual object 2 and the physical object is longer than the distance D2 between the first virtual object 1 and the physical object It is automatically set up so that the user can quickly recognize detailed information.

Also, when a plurality of virtual objects are allocated to a physical object, the distance between the virtual objects is arranged closer to the virtual object, and the distance between the virtual objects is arranged more automatically have.

Figs. 14 and 14A are diagrams showing another operation principle of the augmented reality system 1. Fig.

14 and 14A, the amount of information of a virtual object of a three-dimensional virtual image allocated to and displayed on actual objects of actual image information is automatically dynamically adjusted according to a distance between a real object and a user, ). ≪ / RTI >

Therefore, when the user approaches the object, the virtual object displays more detailed information, so that the user can conveniently check the information of the desired virtual object. That is, the time for the user to concentrate on the virtual object is prolonged, and the information transfer effect can be increased.

In general, the user tends to look closer at the information about the object of interest, so that when the user is far away from the object of interest, the information is abstractly displayed, and when the user is close to the object of interest, .

A virtual object assigned to an actual object starts to be displayed from a time when a distance between a user and a real object or a virtual object reaches within a predetermined separation distance D1 and the closer the distance between the user and the actual or virtual object becomes A virtual object of more detailed information is displayed.

That is, information of a virtual object allocated to one real object is layered. As shown in FIG. 4A,

The most abstract virtual object A is displayed when the user enters a predetermined separation distance D1 and becomes a virtual object having more specific information when the user approaches (D2) the actual or virtual object more A1, A2) are displayed. Also, when the user approaches (D3) closest to the actual object or the virtual object, the virtual objects (A1-1, A1-2, A2-1, A2-2) having the most specific information are displayed.

For example, assuming that a vending machine is placed as a real object in front of the user,

When the user enters the predetermined separation distance D1, the virtual object allocated to the vending machine is displayed. Here, the virtual object is assumed to be the icon of the vending machine.

Next, when the user approaches the vending machine more (D2), the icons of the vending products sold in the vending machine can be displayed as virtual objects of more detailed information.

Finally, when the user closest approaches the vending machine (D3), the calories, ingredients, etc. of each beverage product can be displayed as a virtual object of more detailed information.

As another example, assuming that there is an automobile store as a real object in front of the user,

When the user enters the predetermined separation distance D1, the virtual object allocated to the car dealership is displayed. Here, the virtual object is assumed to be a car brand icon to be sold.

Next, when the user accesses the car dealer more (D2), the various kinds of car icons being sold can be displayed as virtual objects of more detailed information. At this time, a virtual object can be displayed in the vicinity of the currently displayed car, that is, an actual object if there is an actual object, and a virtual object can be displayed in the surrounding area even when the car is not displayed .

Finally, when the user closest approach (D3) to the car dealership, the specification of the vehicle being sold, the price, and the expected date of shipment can be displayed as virtual objects of more detailed information.

On the other hand, when vibration occurs in a virtual object or an actual object that the user wants to check, the mobile terminal 100 calculates a movement distance corresponding to the rate of change of vibration, and then, based on the gazing direction and the calculated movement distance The amount of information of the virtual object can be reset and displayed.

That is, it is assumed that the user has moved by virtual vibration without directly moving, or a weight is given to the movement distance by vibration, and the information amount of the virtual object is reset according to the virtual movement distance and displayed.

That is, the large change rate of the vibration means that the user is running or moving fast, and the small change rate of the vibration corresponds to the user moving slowly, so that the travel distance can be calculated based on this. Therefore, the user may be able to reflect the virtual movement distance by imparting vibration while shaking the head up and down without actually moving.

When the user gazes at a direction to be checked, for example, when the user turns his / her head to the right and continues to vibrate while looking at the right, the mobile terminal 100 detects the moving direction Calculate the travel distance.

That is, the mobile terminal 100 senses the head rotation of the user through the built-in sensor, calculates the virtual current position after detecting the vibration, and grasps the movement distance of the walking or running motion through the rate of change of vibration.

The mobile terminal 100 may be configured to sense the gazing direction based on the direction of rotation of the head when the direction of gazing by the user is sensed, and may be configured to sense the gazing direction by sensing the moving direction of the pupil.

Further, it is possible to simultaneously detect the rotational direction of the head and the moving direction of the pupil, and calculate the gaze direction more precisely by differentiating the specific gravity first from the two detection results. That is, it may be configured to give a specific gravity of 50% to 100% to the rotation angle detection due to the rotation of the head, and give a specific gravity of 0% to 60% to the moving direction of the pupil to calculate the gaze direction.

In addition, the mobile terminal 100 may select and execute the movement distance extension setting mode so that the user can set the distance weight to be 2 to 100 times the calculated virtual movement distance.

In calculating the moving distance corresponding to the rate of change of vibration, the mobile terminal 100 may calculate a moving distance corresponding to the rate of change of vibration by excluding the upper value 10% of the vibration magnitude and the lower value 20% The movement distance can be calculated based on the basis.

As a result, the user can identify a virtual object that has the same amount of information as a real object or a virtual object, even if the user does not actually access the actual object or the virtual object or approaches the virtual object.

15 is a configuration diagram of the mobile terminal 100 of the augmented reality system 1. Fig.

15, the mobile terminal 100 includes a display 110, a left front camera 121, a right front camera 122, a left 3D sensor 131, a right 3D sensor 132, A satellite module 141, a communication module 142, a nine axis sensor 143, a battery 144, a recognition camera 145, and a control unit 150.

The display 110 constitutes a display area of the mobile terminal 100 as a transparent material display.

The left front camera 121 is mounted on the left side of the terminal to acquire actual image information of the front side. Also, the right front camera 122 is mounted on the right side of the terminal to acquire actual image information of the front side.

The left 3D sensor 131 and the right 3D sensor 132 operate in conjunction with the left front camera 121 and the right front camera 122 so as to take a 3D image of the front side. In other words, the photographed 3D image can be stored in the built-in memory or transmitted to the server 200. For reference, the front camera and the 3D sensor may be arranged one by one according to the embodiment to obtain actual image information. It is preferable that the front camera is configured to be able to photograph both the infrared region and the visible light region.

The communication module 142 may be equipped with a Wi-Fi communication module, a Bluetooth communication module, and a broadband (3G, 4G, LTE) communication module .

The 9-axis sensor 143 is referred to as a 9-axis sensor because the values of 9 axes, i.e., 3 axes of acceleration, 3 axes of inertia and 3 axes of geomagnetism are measured, and a temperature sensor may be additionally provided for correcting the temperature value . The 9-axis sensor 143 not only senses the three-dimensional movement of the mobile terminal 100, but also senses the gazing direction, the moving direction, and the tilt of the user.

The battery 144 may be configured to be capable of supplying driving power to the mobile terminal 100 and may be a rechargeable lithium ion battery or a pseudo capacitor.

For reference, the battery 144 may be composed of a plurality of pseudo capacitors. The pseudo capacitors use a two-dimensional oxidation-reduction reaction at the electrodes, The advantage is that the life span is relatively long.

The recognition camera 145 detects the movement of the user's eyes, the gazing direction of the eyes, and the size of the eyes. The recognition camera 145 is most preferably disposed on the left and right sides, respectively, and may be disposed in only one direction. Basically, the recognition camera 145 takes a direction in which the user's eyes are located.

The control unit 150 includes a display 110, a left front camera 121, a right front camera 122, a left 3D sensor 131, a right 3D sensor 132, a satellite module 141, a communication module 142, Axis sensor 143, the battery 144, and the recognition camera 145, as shown in FIG.

Meanwhile, the controller 150 may be configured to selectively charge at least one of the plurality of pseudo-capacitors according to the magnitude of the charging power. The charging method will be described in detail as follows.

It is assumed that when three pseudo-capacitors are arranged, that is, a first pseudo capacitor, a second pseudo capacitor, and a third pseudo capacitor are arranged. It is assumed that the charge capacity of the first pseudo capacitor is the largest, the charge capacity of the second pseudo capacitor is smaller than that of the first pseudo capacitor, and the charge capacity of the third pseudo capacitor is smaller than that of the second pseudo capacitor.

The controller 150 detects the charged amount of the first pseudo capacitor, the second pseudo capacitor, and the third pseudo capacitor, and then supplies the driving power in the order of the highest charge amount.

For example, when the charge amount of the first pseudo capacitor is 60%, the charge amount of the second pseudo capacitor is 70%, and the charge amount of the third pseudo capacitor is 80%

The power of the third pseudo capacitor is supplied first, and when the amount of charge reaches 40%, the power supply of the third pseudo capacitor is interrupted and the power of the second pseudo capacitor is supplied. Further, when the charged amount of the second pseudo capacitor reaches 40%, the power supply of the second pseudo capacitor is interrupted and the power of the first pseudo capacitor is supplied.

If the first to third pseudo-capacitors are all charged at 40% or less, the controller 150 connects the first to third pseudo capacitors in parallel to supply the driving power.

In addition, the augmented reality system 1 may further include a plurality of sensing units 300, each of which includes a Wi-Fi communication module and is disposed at a predetermined interval in a room.

The plurality of sensing units 300 may be selectively arranged to detect the position of the mobile terminal 100 in the room,

Each of the plurality of sensing units 300 transmits the sensing information to the server 200 every time it senses a WIFI HOTSPOT signal periodically output from the mobile terminal 100, The relative position of the mobile terminal 100 can be grasped based on the absolute position of the mobile terminal 100.

As described above, in the proposed system, the present position information is acquired based on the satellite position information in the outdoor, and the current position information is acquired using the Wi-Fi signal in the room.

An additional method for acquiring current position information in indoor and outdoor is described as follows.

The method using the Wi-Fi signal can be basically divided into triangulation and fingerprinting.

First, triangulation measures the received signal strength (RSS) from three or more access points (APs), converts the signal intensity to distance, and calculates the position through the equation.

Next, in the fingerprinting method, the indoor space is divided into small cells, and signal intensity values are directly collected from each cell, and a database is constructed to form a radio map. Then, It estimates the cell that shows the most similar signal pattern compared with the database to the user's location.

Next, a method of collecting position data of each smartphone while exchanging Wi-Fi signals directly or indirectly from a plurality of users having smart phones in the vicinity may be used.

In addition, since the communication module 142 of the mobile terminal 100 includes the Bluetooth communication module, the current location information can be grasped by using the Bluetooth communication.

First of all, there is a method of estimating that a user is located around a beacon when communicating with any one beacon after arranging a plurality of beacons in an indoor space.

Next, a receiver having a plurality of directional antennas arranged on the surface of a hemispherical shape is disposed in an indoor space, and then a user's position is estimated through identification of a specific directional antenna that receives a signal transmitted from the mobile terminal 100 . In this case, when two or more receivers are arranged, the user's position may be identified in a three-dimensional form.

In addition, the current position information of the user is grasped based on the satellite position information, and the velocity and the moving direction are estimated using the information of the 9-axis sensor 143 when entering the shadow area of the satellite position information. That is, the position of the user can be estimated through indoor step counting, stride length estimation, and heading estimation. At this time, in order to increase the accuracy of the estimated information, information such as the user's physical condition (key, weight, stride) may be input and used for the position estimation calculation. When using the information of the 9-axis sensor 143, the accuracy of the estimated information can be improved by combining the position estimation technique using the Wi-Fi communication.

In other words, even if the accuracy is low with the Wi-Fi technique, the absolute position is globally calculated and the relative position of the locally high accuracy is obtained through the information of the 9- Can be combined to improve accuracy. In addition, the accuracy of the position estimation information may be further improved by further applying the Bluetooth communication method.

In addition, since the magnetic field information unique to the position is formed in the indoor space, a method of estimating the current position similarly to the fingerprinting method using Wi-Fi can be applied after constructing the magnetic field map of each space. At this time, the trend of the change in the magnetic field generated when the user moves can be used as additional information.

There is also a method of utilizing the illumination installed in the indoor space. That is, a method of outputting a specific position identifier while blinking the LED illumination at a speed which the person can not identify, and the camera of the mobile terminal 100 recognizing the specific position identifier and estimating the position may be used.

In addition, there is a technique to database the images taken from various locations and various angles of the indoor space, and then to match the pictures taken at the user's location. In addition, various landmarks (signs, trademarks, room numbers, signs, etc.) A method of correcting the position can be used.

For reference, it is most preferable that the method of acquiring the current position information in the indoor space improves the accuracy by combining at least one of the above methods.

In the augmented reality system according to the embodiment of the present invention, when a handwriting is detected along a dotted line guide, a dotted line guide is displayed on a border of a character displayed on a display, and a virtual object corresponding to the character content is displayed .

That is, the user can perform handwriting practice by providing a dotted line guide, and the character font of the dotted line guide is standardized, so that the character recognition rate is high.

In particular, children between 0 and 5 years old can learn a specific character regardless of the character, because they recognize it as an object, that is, a picture or a lump, instead of recognizing a character as an adult. Writing letters can be done by drawing or coloring. In addition, learning can be done using multidisciplinary information in a three-dimensional manner. Children express, develop and express their senses in a way that they see, hear, and feel. In the present invention, it is possible to increase the perception ability through the expression method depending on the senses, thereby enabling sensory immersion for information.

Here, the sensory immersion refers to how much the user's attention falls into the information of the virtual world before the eyes. For example, when the object is augmented in augmented reality, the children concentrate and observe the complete image of the object Touch the screen with your finger or move the camera. By providing a real-world interface that allows children to manipulate virtual objects through specific drawing (writing activities) activities, it can induce natural behavior in learning situations and immerse themselves in learning itself.

In addition, since the attitude of at least one or more virtual objects selected in the augmented reality system according to the embodiment of the present invention is changed in synchronization with the three-dimensional attitude information of the mobile terminal, the attitude of the virtual object can be changed variously with a minimum of actions.

In addition, the selected virtual object can be moved / rotated three-dimensionally using the three-dimensional movement / rotation information of the mobile terminal. That is, the virtual object can be easily moved / rotated by changing the attitude of the mobile terminal without touching the touch screen (display) many times without converting the moving and rotating operation modes.

In addition, it is possible to record the motion of the virtual object, to store and reproduce the movement of the virtual object, and to improve the utilization such as the augmented reality animation implementation.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: mobile terminal
200: Server
300: sensing part
110: Display
121: Left front camera
122: Right front camera
131: Left 3D sensor
132: Right 3D sensor
141: Satellite module
142: Communication module
143: 9 axis sensor
144: Battery
145: Recognition camera
150:

Claims (6)

In displaying a three-dimensional virtual image on a display, a dotted line guide is displayed on a border of a character displayed on the display, and when a handwriting is sensed along the dotted line guide, The mobile terminal comprising:
Wherein when the virtual object is touched, movement of a predetermined virtual object corresponding to a touched portion is reproduced.
The method according to claim 1,
Wherein the mobile terminal comprises a 9-axis sensor to acquire its own 3-dimensional attitude information, and the attitude of at least one or more virtual objects selected in the mobile terminal is changed in synchronization with the 3-dimensional attitude information. .
A method for displaying a three-dimensional virtual image on a display, the method comprising the steps of: displaying the current position information and the three-dimensional virtual image corresponding to the actual image information on the display; The mobile terminal comprising:
Wherein the mobile terminal displays a character corresponding to a name of the 3D virtual image on the display, a dotted line guide is displayed on an edge of the character, and when the handwriting is detected along the dotted line guide, Displaying a virtual object corresponding to the character content,
Wherein when the virtual object is touched, movement of a predetermined virtual object corresponding to a touched portion is reproduced.
The method of claim 3,
Wherein the mobile terminal comprises a 9-axis sensor to acquire its own 3-dimensional attitude information, and the attitude of at least one or more virtual objects selected in the mobile terminal is changed in synchronization with the 3-dimensional attitude information. .
A method for displaying a three-dimensional virtual image on a display, the method comprising the steps of: displaying the current position information and the three-dimensional virtual image corresponding to the actual image information on the display; A mobile terminal; And
And a server for providing the mobile terminal with the current location information transmitted from the mobile terminal and the 3D virtual image corresponding to the actual image information in real time,
Wherein the server identifies a real object of the real image information and provides the virtual object of the three-dimensional virtual image assigned to the identified real object to the mobile terminal,
Wherein the mobile terminal displays a character corresponding to a name of the 3D virtual image on the display, a dotted line guide is displayed on an edge of the character, and when the handwriting is detected along the dotted line guide, Displaying a virtual object corresponding to the character content,
When the virtual object is touched, a predetermined movement of the virtual object corresponding to the touched portion is reproduced,
Wherein the mobile terminal comprises a 9-axis sensor to acquire its own 3-dimensional attitude information, and the attitude of at least one or more virtual objects selected in the mobile terminal is changed in synchronization with the 3-dimensional attitude information. .
6. The method of claim 5,
Wherein coordinates of at least one or more virtual objects selected in the mobile terminal are changed from a spatial coordinate system for the actual image information to a mobile coordinate system of the mobile terminal, and coordinates of the deselected virtual objects are changed in the mobile coordinate system of the mobile terminal And the spatial coordinate system for the image information.

Images