JP2011060254A - Augmented reality system and device, and virtual object display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an augmented reality system for displaying a virtual object input by a user to oneself. <P>SOLUTION: In the augmented reality system, when the user inputs the virtual object, information regarding a position and a direction of the augmented reality device is obtained, and the obtained information regarding the position and the direction is recorded together with the information of the virtual object. When the user uses the augmented reality system, the virtual object input by the user oneself is superposed on a real environment and displayed in approaching the position and the direction regarding the virtual object input by the user oneself. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to augmented reality technology in portable devices.

  In a portable device in which a camera device and a display device are mounted, there is a technique for displaying virtual image data prepared in advance for an image photographed by a camera device and projected on a display device in real time (for example, , See Patent Document 1). According to such a technique, it is possible to superimpose mechanically created image data on information given to perception from an actual environment, and to provide supplementary information to actual visual information.

  Patent publication 2008-510254

  However, in the technology for displaying image data prepared in advance on an actual environment, it is assumed that the image data prepared in advance is displayed. It was necessary to generate and store in a portable device in advance. Also, image data has a relatively large data capacity compared to a text file or the like. In order to express the augmented reality environment in every scene, it is necessary to prepare a large number of image data, and it is necessary to secure a large storage area in order to store the image data. Furthermore, while image data is a virtual object that is good at visual expression, it is not good at expressing text information that is useful in augmented reality. Therefore, the present invention provides that the user himself / herself generates a virtual object on the real environment displayed on the display device, accumulates the virtual object together with the position data of the real environment, and the virtual object as needed. The augmented reality technology that calls the position data and displays it on the actual environment is realized.

  The augmented reality system according to the present invention includes a first acquisition unit that acquires an image in an actual environment, a second acquisition unit that acquires a position marker indicating a reference position for acquiring the image, and a reference direction for acquiring the image. A third acquisition unit that acquires a direction marker that indicates a virtual object corresponding to the position marker acquired by the second acquisition unit and the direction marker acquired by the third acquisition unit; Acquisition means, recording means for recording the virtual object as related information of the position marker and the direction marker, and display means for displaying the virtual object superimposed on the image. According to this configuration, the user himself / herself can generate a virtual object corresponding to the position and direction from which the image was acquired, and the virtual object corresponding to the position and direction from which the image was acquired can be superimposed on the image It is. By doing so, the user can arrange any virtual object at any augmented reality position. The virtual object may be text information, for example, in which case the information capacity is small.

  In the augmented reality system according to the present invention, the second calculation unit repeats the calculation of the second position marker according to a change in the position at which the image is acquired, and the third calculation unit includes the The calculation of the third direction marker is repeated according to a change in the direction of acquiring an image, and if there is a request for generation of the virtual object, the second position marker calculated by the second calculation unit and It is possible to adopt a configuration in which the third direction marker calculated by the third calculation means is recorded and the position and direction in which the virtual object is input are acquired. According to this configuration, the position and direction in which the virtual object is input are uniquely determined with respect to the position and direction in which the image is acquired by the first acquisition unit. It is possible to display the virtual object so as to overlap the image under a condition that matches the position and direction in which the virtual object is input.

  In the augmented reality system according to the present invention, since the generation of the virtual object, the generation of the position marker, and the generation of the direction marker are performed simultaneously, the position marker and the direction marker from which the virtual object is acquired are displayed. It can be recorded as related information of the virtual object. By simultaneously recording the position marker and the direction marker as related information of the virtual object, when a predetermined condition is satisfied, the accumulated virtual object is displayed on the image of the first real environment. It is displayed in a superimposed manner.

  The program according to the present invention includes a step of acquiring an image, a step of acquiring a position marker indicating a reference position for acquiring the image, and a direction marker indicating an acquisition direction of the image from the reference position indicated by the position information. Acquiring a virtual object based on a request from a user at an arbitrary position and direction, recording the virtual object together with the position marker and the direction marker, and acquiring the image A step of superimposing and displaying the virtual object on the image when the virtual object approaches a position and a direction that coincide with the position and direction recorded at the same time.

  According to the present invention, in an augmented reality system, when an image is acquired, a virtual object input in association with position information and direction information is recorded, and when the user acquires an image next time, the position information According to the direction information, the previously recorded virtual object can be displayed superimposed on the real image.

  Diagram showing configuration of augmented reality system   Block diagram showing configuration of augmented reality device   Database configuration diagram in database server   Diagram showing the configuration of the shooting device   Diagram showing configuration of input device and display device   Diagram showing input of virtual object   Diagram showing virtual object display

[Embodiment]
FIG. 1 is a diagram showing a configuration of an augmented reality system according to an embodiment of the present invention. As shown in the figure, the augmented reality system of the present embodiment includes a plurality of augmented reality devices 100 and a database server 10. The augmented reality device 100 and the database server 10 are communicably connected via the network NW1. The augmented reality device 100 of the present embodiment has a portable structure, and a typical example is a mobile phone information terminal (that is, a wireless communication terminal). In this case, the network NW1 includes a wireless mobile communication network.

  FIG. 2 is a block diagram showing the configuration of the augmented reality device in the present embodiment. As shown in the figure, the augmented reality device 100 includes a control unit 110, a communication unit 120, a photographing device 130, a direction sensor 140, a position sensor 150, an input device 160, and a display device 170.

  The control unit 110 includes an arithmetic processing device such as a CPU (Central Processing Unit) and a memory, and controls the operation of each unit of the augmented reality device 100 by executing a program.

  The communication unit 120 includes an interface for communicating with the network NW1, and transmits / receives data to / from the database server 10. The data transmitted and received by the communication unit 120 includes augmented reality device identification information that the device itself has, virtual object data in the device, position markers, and direction markers.

  The augmented reality device identification information that the device itself has is prepared for the database server to manage a plurality of augmented reality devices. The augmented reality device identification information includes unique information that is different for each augmented reality device, and the information includes a character string, a number, or a symbol. Thus, in the augmented reality system, a plurality of augmented reality devices can operate simultaneously, and each augmented reality device can manage its own data.

  The photographing device 130 will be described with reference to FIG. The imaging device 130 is mounted on a surface different from the surface on which the display device 170 described later is mounted in the augmented reality device. In a typical mobile phone information terminal, the shooting device 130 is mounted on the back surface of the mobile phone information terminal. The The photographing device 130 has a photographing function for photographing an image or video, and the photographed image or video is supplied to the control unit 110 and can be displayed on a display device 170 described later in real time. .

  The direction sensor 140 includes a sensor that detects a shooting direction from a position serving as a reference for shooting. The direction sensor 140 includes at least a gravity sensor that detects the gravity of the earth, and includes a geomagnetic sensor that detects an azimuth based on geomagnetism, and a sensor that detects displacement and angular velocity. The direction sensor 140 supplies a direction marker indicating the shooting direction to the control unit 110. The direction marker indicates at least the shooting direction.

  The position sensor 150 receives information for acquiring position information indicating a position serving as a reference for photographing and supplies the information to the control unit 110. The position information of the present embodiment is information indicating the position by latitude and longitude. The position sensor 150 of the present embodiment may receive radio waves from a so-called GPS (Global Positioning System) satellite. Alternatively, identification information may be received from a communication device (information on a base station of a mobile communication network or an access point of a wireless LAN).

  The input device 160 will be described with reference to FIG. The input device 160 is a position input device in which a user operates a device by pressing a display on the panel, and is configured by a sensor panel such as colorless and transparent glass or acrylic. The input device 160 senses and supplies information on the position of the touched sensor panel region to the control unit 110 when the user touches a point or region with his / her hand.

  The display device 170 will be described with reference to FIG. The display device 170 is a display device represented by a liquid crystal display device or the like. The display device 170 is a two-layer system with the input device 160, and the upper layer is configured as the input device 160 and the lower layer is configured as the display device 170. By touching a point or region such as a picture or image displayed on the display device 170 with a hand, information on the region position can be supplied to the control unit 110 through the input device 160. In FIG. 5, the positions of the display input 170 and the input device 160 are shifted from each other, but this is to show that two different devices are mounted. It is piled up. The display device 170 can display an image or video captured by the imaging device 130 through the control unit 110 in real time.

  The structure of the database in the database server will be described with reference to FIG. The database in the database server includes augmented reality device identification information, virtual object data, a position marker, and a direction marker. The augmented reality device identification information is information assigned to each augmented reality device, and is used to specify which augmented reality device is recorded information. The virtual object data is information on a character string input by a method described later. The position marker is information regarding the measurement position obtained from the position sensor 150. The direction marker is information on the measurement direction obtained from the direction sensor 140. In this database, in response to a write request from the augmented reality device 10, responding to the addition of virtual object data while keeping the items of augmented reality device identification information, virtual object data, position marker, and direction marker constant. Can do. Further, in this database, in response to a read request from the augmented reality device 10, only the element corresponding to the augmented reality device identification number of the augmented reality device 10 that has requested read is extracted, and only the corresponding augmented reality device 10 is extracted. It is possible to send.

  The configuration of the augmented reality system of this embodiment is as described above. Under this configuration, the user starts using the augmented reality device 100 at an appropriate position outdoors or indoors. The augmented reality device 100 executes the following recording process (hereinafter referred to as “virtual object recording process”) in response to such an operation, and records virtual object data, a position marker, and a direction marker in the recording database server 10.

  A method of generating a virtual object in the virtual object recording process will be described with reference to FIG. In FIG. 6, when the user inputs characters through the character input keypad 200, the input characters are displayed on the input console 210, and the character string input by the user can be confirmed. The user creates a character string by repeating character input, and presses the enter key on the input keypad 200 to confirm the input content. The character string determined by this work becomes virtual object data. The virtual object data is transmitted to the database server 10 through the communication unit 120 together with the direction marker obtained from the direction sensor 140 and the position marker obtained from the position sensor 150 when the virtual object data is generated. At this time, the extended identification device information is transmitted simultaneously.

  Based on the information received from the augmented reality device 100, the database server 10 records the augmented identification device information, virtual object data, position markers, and direction markers in the database. At this time, information indicating that the recording has been normally performed on the database may be transmitted to the corresponding augmented reality device. Thus, the virtual object recording process is completed.

  The augmented reality device 100 executes the following virtual reality display processing (hereinafter referred to as “virtual object display processing”) in accordance with a user operation, and records virtual object data, position markers, and direction markers. The virtual reality display is realized by reading out from.

  The virtual object display process will be described with reference to FIG. In the augmented reality apparatus 100, in a state where an image or video captured by the imaging device 130 is displayed on the display device 170 in real time, the direction sensor 140 positions information regarding the current position of the augmented reality apparatus 10 and positions information regarding the current position. The sensor 150 acquires and transmits to the control part 110 for every fixed time. When an acquisition request for information related to the device is sent from the augmented reality device 100 to the database server 10, virtual object data, position markers, and direction markers related to the corresponding augmented reality device recorded in the database in the database server 10 are displayed. And transmitted to the augmented reality device 100 that has made the acquisition request. The communication unit 120 in the augmented reality device 100 transmits the data group received from the database server 10 to the control unit 110.

  The control unit 110 determines a virtual object to be displayed on the display device 170 and determines at which position in the display device the virtual object 300 is displayed. Based on the position marker and the direction marker transmitted from the database server 10, a point serving as a reference for the position and direction of the own apparatus when the virtual object recording process is executed is determined as the virtual object position P1 in FIG. Although the virtual object position P1 is clearly shown in FIG. 7 for explanation, it is not actually visible to the user. The control unit 110 controls the display of the virtual object position P1 based on the information on the current position and the information on the current direction obtained from the direction sensor 140 and the position sensor 150. The virtual object position P1 moves within the display device 170 in accordance with the movement of the own apparatus. However, since the virtual object position P1 is spatial information that can be expressed in three dimensions indicated by the position and direction, the control unit 110 does not use the real environment. A virtual object position P1 corresponding to the position at is specified in the display device 170.

  The control unit 110 displays the virtual object data recorded when the virtual object recording process is performed, overlapping the virtual object position P1 on the display device 170. As shown in FIG. 7, the virtual object data recorded as a character string is displayed on the display device 170 as the virtual object 300. Since the virtual object 300 is displayed in a fixed positional relationship with the virtual object position P1, when the virtual object position P1 moves in the display device 170 in accordance with the movement of the augmented reality device 100, the same movement amount and The virtual object 300 also moves in the moving direction. As a result, the virtual object data input by the user can be displayed as an augmented reality on the real environment. Thus, the virtual object display process is completed.

(Modification 1)
In the virtual object recording process, in addition to inputting a character string, the virtual object 300 may be implemented by embedding a handwritten pen drawn through the input device 160, an image or video recorded through the photographing device 130, and the like. In this case, also in the virtual object display process, the virtual objects recorded for the real environment are displayed.

(Modification 2)
In the virtual object display process, a plurality of virtual objects 300 may be displayed in the display device 170 at the same time.

(Modification 3)
A function corresponding to the database server 10 is provided in the own apparatus instead of the communication unit 120, and the virtual object recording process is performed in the own apparatus without causing communication between the database server 10 and the own apparatus through the network NW1. Virtual object display processing may be performed.

DESCRIPTION OF SYMBOLS 10 ... Database server, 100 ... Augmented reality apparatus, 110 ... Control part, 120 ... Communication part, 130 ... Shooting device, 140 ... Direction sensor, 150 ... Position sensor, 160 ... Input device, 170 ... Display device, 200 ... Character input Keypad, 210 ... input console, 300 ... virtual object

Claims (2)

Means for acquiring real-world image data;
Means for generating a position marker corresponding to the acquisition position of the image data;
Means for generating a direction marker corresponding to the acquisition direction of the image data;
Means for generating a virtual object corresponding to the position marker and the direction marker;
Means for recording the virtual object with the position marker and the direction marker as related information;
Means for calling the recorded virtual object at a corresponding position and direction of the position marker and the direction marker, and superimposing the virtual object on real environment image data;
Augmented reality system comprising: The augmented reality system, wherein the virtual object records user input data including at least one of text input data, pen input data, voice input data, and photographed image data.

Images