JP6475776B2 - Augmented reality system and augmented reality providing method - Google Patents

Description

  The present invention relates to a technology for realizing augmented reality by extending a real environment.

  2. Description of the Related Art Conventionally, a technology for realizing augmented reality (AR) that augments a real environment by adding information created by a computer or the like to information in the real environment is known. Augmented reality is generally given visually. Therefore, in order to realize augmented reality, it is necessary for the computer to know the field of view of the user experiencing the augmented reality, and it is necessary to grasp the position of the user.

  Therefore, an augmented reality technique using GPS (Global Positioning System) has been proposed as a technique for grasping the position. For example, the technique described in Patent Document 1 determines the content of a virtual object to be displayed and the display position in a real environment based on position information acquired by GPS, and obtains real visual information and a virtual object. This is a composite display technology. In patent document 1, the surrounding feature information is acquired from the position information acquired from GPS from a database, and the virtual environment is drawn on the transmissive display to realize the synthesis of the real environment and the virtual object.

  Also, for example, augmented reality technology using captured images is also known. Augmented reality technology using photographed images installs an image (marker) having dedicated identification information in real space, and if a marker exists in the photographed image, a method of drawing a virtual object on the marker, There is a method of recognizing a specific figure (such as a human body) and drawing a virtual object. For example, Patent Document 2 describes a technique that enables a virtual object to be superimposed and displayed using a marker that exists in a real environment. Japanese Patent Application Laid-Open No. 2004-228561 describes a technique that enables display of a virtual object by identifying a known real space arrangement without requiring a marker as a real object in the real environment.

JP 2012-068481 A JP 2012-141777 A JP 2003-256876 A

  However, the technique described in Patent Document 1 has a problem that the display position of the virtual object is shifted due to a decrease in GPS positioning accuracy in an environment where the GPS signal is weak. In the first place, the technique described in Patent Document 1 has a problem that even a virtual object cannot be displayed in an environment where GPS signals cannot be received. In particular, augmented reality is usually provided indoors, and indoors have a characteristic that GPS signals tend to be weak. Therefore, there is a situation where augmented reality and GPS are not compatible. Augmented reality is realized by a portable output device. However, when using a GPS signal, the GPS signal must always be received and processed, which is consumed by a portable output device that is forced to be driven by a battery. There is also a problem of increasing the power.

  In addition, the techniques described in Patent Documents 2 and 3 have a problem that recognition may fail depending on the state of the marker or the arrangement. In addition, it is necessary to install a specific marker or arrangement in the real space, and there is a problem that the landscape is damaged. In addition, when changing the display content of the virtual object, there is a problem in that the versatility is deteriorated because it is necessary to move the marker or the arrangement object in the real environment. Furthermore, since an image recognition process is always required for a captured image while augmented reality is being executed, there is a problem that the amount of calculation is large, and the load on the computer and the power consumption are large.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for realizing augmented reality without requiring a dedicated marker even in an environment where GPS signals cannot be received. To do.

In order to solve the above-mentioned problem, the invention of claim 1 is an augmented reality system comprising a mobile terminal device whose position is variable and an index means whose absolute position is known, and a sensor for measuring information relating to movement; A first storage means for storing a reference position of the sensor and an attitude of the sensor at the reference position; an acquisition means for acquiring individual information of the index means; and whether the sensor is present at the reference position A determination means for determining the reference position stored in the first storage means after the determination means determines that the sensor is present at the reference position, and the measurement by the sensor. A position specifying unit that specifies the current position of the sensor based on information relating to movement; and after the determination unit determines that the sensor is present at the reference position And an attitude specifying means for specifying the current attitude of the sensor based on the attitude of the sensor at the reference position stored in the first storage means and information on the movement measured by the sensor; An output means for expressing augmented reality by outputting output information according to the current position of the sensor specified by the position specifying means and the current posture of the sensor specified by the attitude specifying means; And the indicator means defines the orientation of the sensor when the sensor is present at the reference position to be the orientation of the sensor at the reference position stored in the first storage means. comprising a body, said sensor and said acquisition means, said provided in the portable terminal apparatus, the determining means, number of the index means by the acquisition unit It determines that said sensor when the information is acquired is present in the reference position.

  The invention of claim 2 is an augmented reality system according to the invention of claim 1, comprising a plurality of the mobile terminal devices, and the mobile terminal device is another mobile terminal device existing in an augmented reality space. A second communication unit that performs wireless data communication with the mobile station, and the output unit, wherein the second communication unit receives the current position of the sensor included in the other mobile terminal device by wireless data communication. Then, the output means outputs output information according to a current position of the sensor included in the other portable terminal device received by the second communication means.

  The invention of claim 3 is the augmented reality system according to the invention of claim 1, wherein the mobile terminal device performs wireless data communication with a terminal device existing in the augmented reality space. The terminal device includes communication means and the output means, wherein the third communication means receives specific information regarding the terminal device by wireless data communication, and the output means receives the third communication means. Output information according to the specific information about.

  The invention according to claim 4 is the augmented reality system according to any one of claims 1 to 3, further comprising second storage means for storing information relating to the object to which the sensor is attached, wherein the output The means outputs output information corresponding to information on the object stored in the second storage means.

  The invention according to claim 5 is the augmented reality system according to any one of claims 1 to 4, further comprising a biological sensor for measuring biological information related to a living body, wherein the output means is configured by the biological sensor. Output information corresponding to the measured biological information is output.

Further, the invention of claim 6 is a method for providing augmented reality by a mobile terminal device whose position is variable, the step of measuring information relating to movement by a sensor, the reference position of the sensor, and the attitude of the sensor at the reference position Storing in the first storage means, obtaining the individual information of the indicator means whose absolute position existing outside is known, and the attitude of the sensor when the sensor is present at the reference position, A step of defining by a housing of the indicator means so as to be an attitude of the sensor at the reference position stored in the first storage means, and determining whether the sensor is present at the reference position A step, and after the determination that the sensor is present at the reference position, the reference position stored in the first storage means and the measurement by the sensor The step of identifying the current position of the sensor based on the information relating to the movement, and after determining that the sensor is present at the reference position, the information stored in the first storage means The step of identifying the current posture of the sensor based on the posture of the sensor at a reference position and the information on the movement measured by the sensor, and the sensor identified as the current position of the identified sensor Outputting augmented reality by outputting output information according to the current posture of the sensor, and when the individual information of the indicator means is acquired, in the determining step, the sensor It is determined that it exists at the position.

The invention according to claims 1 to 5, a first storage means for storing a sensor for measuring information about the motion, the posture of the ruse capacitors put in the reference position and standards position of the sensor, a separate indicator means An acquisition means for acquiring information, a determination means for determining whether or not the sensor is present at the reference position, and after the determination means determines that the sensor is present at the reference position, the first storage means After determining that the sensor is present at the reference position by the position specifying means for specifying the current position of the sensor based on the stored reference position and the information on the movement measured by the sensor, and the determining means A position specifying means for specifying the current position of the sensor on the basis of the position of the sensor at the reference position stored in the first storage means and information on the movement measured by the sensor; And output means for representing the augmented reality by outputting the output information in accordance with the current posture of the sensor specified by the current position and the posture identifying means of the sensor specified by means, indicator means, sensor A housing is provided that defines the orientation of the sensor when present at the reference position to be the orientation of the sensor at the reference position stored in the first storage means. It is determined that the sensor is present at the reference position when the information is acquired. Thereby, even in an environment where GPS signals cannot be received, augmented reality can be realized without installing a marker or the like.

According to the sixth aspect of the present invention, the step of measuring information relating to movement by the sensor, the step of storing the reference position of the sensor and the attitude of the sensor at the reference position in the first storage means, and the absolute position existing outside Acquiring the individual information of the known index means, and the index means so that the orientation of the sensor when the sensor is present at the reference position becomes the orientation of the sensor at the reference position stored in the first storage means Stored in the first storage means after it is determined that the sensor is present at the reference position, the step of determining whether the sensor is present at the reference position, and the step of determining whether the sensor is present at the reference position. The step of identifying the current position of the sensor based on the measured reference position and information on the movement measured by the sensor, and after determining that the sensor is present at the reference position. The step of identifying the current posture of the sensor based on the posture of the sensor at the reference position stored in the first storage means and the information related to the movement measured by the sensor, and the current position of the identified sensor, Outputting augmented reality by outputting output information according to the current posture of the identified sensor, and when the individual information of the index means is acquired, the sensor determines the reference It is determined that it exists at the position. Thereby, even in an environment where GPS signals cannot be received, augmented reality can be realized without installing a marker or the like.

It is a figure which shows the augmented reality system in 1st Embodiment. It is a block diagram of the portable terminal device, reference position acquisition apparatus, and database server in 1st Embodiment. It is a figure which shows the functional block with which the portable terminal device in 1st Embodiment is provided with the flow of data. It is a flowchart which shows the augmented reality provision method in 1st Embodiment. It is a figure which shows the example of a display of the landscape of the augmented reality space provided with respect to a user in 1st Embodiment. It is a figure which shows the example of a display of the landscape of the augmented reality space provided with respect to a user in 1st Embodiment. It is a figure which shows the augmented reality system in 2nd Embodiment. It is a figure which shows the functional block with which the portable terminal device in 2nd Embodiment is provided with the flow of data. It is a figure which shows the example of the augmented reality implement | achieved by the radio | wireless communication between portable terminal devices. It is a figure which shows the example of the augmented reality implement | achieved by the wireless communication between a portable terminal device and a terminal device. It is a figure which shows the augmented reality system in 3rd Embodiment. It is a block diagram of the portable terminal device in 3rd Embodiment. It is a figure which shows the functional block with which the portable terminal device in 3rd Embodiment is provided with the flow of data. It is a flowchart which shows the augmented reality provision method in 3rd Embodiment. It is a figure which shows the example of the augmented reality implement | achieved by the augmented reality system in 3rd Embodiment. It is a figure which illustrates the display position of the ghost image in augmented reality space. It is a figure which shows the example of a display of a ghost image. It is a figure which shows the example of a display of a ghost image. It is a figure which shows the example of a display of a ghost image. In 3rd Embodiment, it is a figure which shows the modification in the case of changing the display of a ghost image. It is a display example of the augmented reality of the search application provided by the augmented reality system in 4th Embodiment. It is a display example of the augmented reality of the search application provided by the augmented reality system in 4th Embodiment.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. However, unless otherwise specified in the following description, descriptions of directions and orientations correspond to the drawings for the convenience of the description, and do not limit, for example, a product, a product, or a scope of rights.

<1. First Embodiment>
FIG. 1 is a diagram illustrating an augmented reality system 1 according to the first embodiment. The augmented reality space 9 shown in FIG. 1 conceptually illustrates an area where augmented reality by the augmented reality system 1 is provided.

  The augmented reality system 1 includes a portable terminal device 2, a reference position acquisition device 10 configured as an installation type device whose absolute position is known and whose position is fixed to the absolute position, and a database server 11. . The numbers of the mobile terminal device 2, the reference position acquisition device 10, and the database server 11 are not limited to the numbers shown in FIG.

  In the augmented reality system 1 in the first embodiment, the reference position acquisition device 10 and the database server 11 are assumed to be devices provided and installed by a system operator. On the other hand, the portable terminal device 2 is assumed to be a device owned by a user who visits an area where the system operator wants to provide augmented reality, and includes a personally owned mobile phone, smartphone, PDA terminal, etc. To do.

  FIG. 2 is a block diagram of the mobile terminal device 2, the reference position acquisition device 10, and the database server 11 in the first embodiment.

  The mobile terminal device 2 includes a CPU 20, a storage device 21, an operation unit 22, a display unit 23, a sensor group 24, an imaging unit 25, a non-contact IC card unit 26 and a communication unit 27. The mobile terminal device 2 is configured as a device (device whose position is variable) that moves with the user as a target object when the mobile terminal device 2 is carried by the user. In addition, since the mobile terminal device 2 includes the sensor group 24, the sensor group 24 is also attached to the user as the object.

  The CPU 20 reads and executes the program 210 stored in the storage device 21, and performs various data calculations, control signal generation, and the like. Thereby, CPU20 has a function which calculates and produces various data while controlling each structure with which the portable terminal device 2 is provided. That is, the mobile terminal device 2 is configured as a general computer.

  The storage device 21 provides a function of storing various data in the mobile terminal device 2. In particular, the storage device 21 in the present embodiment is used to store the program 210, the reference information 103, the candidate information 112, the measurement information 212, the position information 214, the output information 215, the captured image information 213, and the owner information 211. The

  As the storage device 21, a RAM or buffer used as a temporary working area of the CPU 20, a read-only ROM, a non-volatile memory (such as a NAND memory), a hard disk for storing a relatively large amount of data, a dedicated memory A portable storage medium (a CD-ROM, a PC card, an SD card, a USB memory, or the like) mounted on the reading device is applicable. In FIG. 1, the storage device 21 is illustrated as if it were one structure. However, the storage device 21 is usually composed of a plurality of types of devices that are employed as necessary among the various devices (or media) exemplified above. That is, the storage device 21 is a generic name for a group of devices having a function of storing data (the same applies to storage devices 101 and 110 described later).

  The actual CPU 20 is an electronic circuit provided with a RAM that can be accessed at high speed. However, the storage device included in the CPU 20 is also included in the storage device 21 for convenience of explanation. That is, in the present embodiment, description will be made assuming that the storage device 21 also stores data temporarily stored in the CPU 20 itself.

  The operation unit 22 is hardware that is operated by a user to input an instruction to the mobile terminal device 2 (augmented reality system 1). Examples of the operation unit 22 include various keys and buttons, a touch panel, and a pointing device.

  The display unit 23 is hardware having a function of outputting various data by displaying them. Examples of the display unit 23 include a lamp, an LED, a liquid crystal display, a liquid crystal panel, and the like. In particular, the display unit 23 in this embodiment includes a liquid crystal display that displays an image on a screen, and has a function of expressing augmented reality by outputting output information 215. That is, the display unit 23 in the first embodiment corresponds to an output unit according to the present invention.

  The sensor group 24 includes a plurality of sensors that measure information related to movement. The sensors included in the sensor group 24 include detection devices for executing relative positioning, such as acceleration sensors, gyro sensors, and geomagnetic sensors. An output (measured value) from the sensor group 24 is transferred to the storage device 21 and stored as measurement information 212. Although details will be described later, the CPU 20 calculates a movement path by “movement” based on the measurement information 212.

  The movement path calculated based on the measurement information 212 measured by the sensor group 24 is strictly the movement path of the sensor group 24. However, as already explained, in the augmented reality system 1 according to the present embodiment, when the user carries the mobile terminal device 2, the sensor group 24 is attached to the user as the object. Thereby, the sensor group 24 mainly measures information reflecting the movement of the user. Therefore, the augmented reality system 1 regards the movement path of the sensor group 24 as the movement path of the user associated with the sensor group 24. In the following description, unless otherwise specified, the movement route of the sensor group 24 and the movement route of the object are not distinguished from each other and are simply referred to as “movement route”.

  In addition, it is good also as employ | adopting a prior art and correct | amending and correcting suitably the movement path | route of the sensor group 24, and it is good also as a movement path | route of a more accurate target object. For example, the user's movement route while the measurement information 212 representing the user's walking state is obtained is not calculated using the measurement information 212 but information such as the user's average stride and walking speed (user Information) may be employed for calculation. Further, the sensors included in the sensor group 24 are not limited to the examples shown above.

  The imaging unit 25 includes an optical element such as a lens and a photoelectric conversion element such as a CCD, and has a function of imaging a subject within the imaging range and acquiring captured image information 213 that represents the actual appearance of the subject. Have. That is, the imaging unit 25 has a configuration and a function as a general digital camera.

  Although details will be described later, in the first embodiment, the display unit 23 includes captured image information 213 that represents the actual appearance of a subject that actually exists in the surroundings, and things (characters and the like) that do not exist in the surroundings. The augmented reality is represented on the screen by combining and displaying the output information 215 selected from the candidate information 112 representing the). In the following description, unless otherwise specified, the captured image information 213 is a color moving image composed of a plurality of frame images.

  The non-contact IC card unit 26 has the configuration and functions of a general non-contact IC card. Thereby, the portable terminal device 2 can perform close proximity wireless communication with the non-contact IC card reader unit 100 of the reference position acquisition device 10. It should be noted that conventional techniques (such as various standards) can be appropriately employed as the circuit configuration and functions of the non-contact IC card unit 26. Therefore, a detailed description of the circuit configuration and functions of the non-contact IC card unit 26 is omitted.

  Thus, since the mobile terminal device 2 includes the non-contact IC card unit 26, the user holds the mobile terminal device 2 close to the non-contact IC card reader unit 100 of the reference position acquisition device 10, Necessary information can be acquired from the reference position acquisition device 10 to the non-contact IC card unit 26 side. In particular, the mobile terminal device 2 in the present embodiment acquires the reference information 103 and the candidate information 112 from the reference position acquisition device 10. In the following description, a series of operations in which the user holds the mobile terminal device 2 close to the non-contact IC card reader unit 100 is referred to as “communication enabling operation”.

  The communication unit 27 provides a function in which the mobile terminal device 2 performs wireless communication with an external device. The communication provided by the communication unit 27 is not limited to data communication, and may be a call.

  The reference position acquisition device 10 is a device installed in the vicinity of an area that provides augmented reality. The reference position acquisition device 10 has a known absolute position and is configured as an installation type device that is fixed to the absolute position. As shown in FIG. 2, the reference position acquisition device 10 includes a non-contact IC card reader unit 100 and a storage device 101. In FIG. 2, the detailed configuration of the reference position acquisition device 10 is not illustrated, but the reference position acquisition device 10 includes a CPU, an operation unit, a display unit, a communication unit, and the like. It is configured as a computer.

  The non-contact IC card reader unit 100 performs proximity wireless communication with a general non-contact IC card and can read various information stored in the non-contact IC card. It is also possible to transmit various information to the contact IC card. As such a non-contact IC card reader unit 100, since a conventional technique can be applied, detailed description is omitted. The non-contact IC card reader unit 100 in the present embodiment performs close proximity wireless communication with the non-contact IC card unit 26 provided in the mobile terminal device 2.

  As shown in FIG. 1, the casing constituting the outer surface of the reference position acquisition device 10 has an appearance suitable for the user to perform the communication enabling operation. In other words, the appearance is such that the position and orientation of the mobile terminal device 2 (sensor group 24) when the user performs the communication enabling operation are clearly defined. Specifically, at the position of the non-contact IC card reader unit 100, the outer surface of the housing has a planar shape inclined with respect to the horizontal plane, and the color of the outer surface is different from the other parts. It has been designed. As a result, the user can accurately execute the communication enabling operation without being confused.

  The position and orientation of the mobile terminal device 2 when the user is enabling communication are defined by the casing of the reference position acquisition device 10 as described above. In addition, since the absolute position of the reference position acquisition device 10 is known and the reference position acquisition device 10 is an installation type device, the absolute position does not easily change. Therefore, when the non-contact IC card reader unit 100 of the reference position acquisition device 10 and the non-contact IC card unit 26 of the mobile terminal device 2 perform data communication with each other, the mobile terminal device 2 (sensor group 24). The position and orientation of can be regarded as known.

  The augmented reality system 1 according to the present embodiment is a portable terminal when the non-contact IC card reader unit 100 of the reference position acquisition device 10 and the non-contact IC card unit 26 of the portable terminal device 2 perform data communication with each other. The position of the device 2 (sensor group 24) is referred to as “reference position”, and the posture (orientation) of the sensor group 24 at the reference position is referred to as “posture at the reference position”.

  Both the reference position and the posture at the reference position can be measured in advance for each reference position acquisition device 10 when the reference position acquisition device 10 is installed, and stored as reference information 103. That is, the reference information 103 corresponds to individual information of the reference position acquisition device 10, and the position of the sensor group 24 when the non-contact IC card reader unit 100 and the non-contact IC card unit 26 perform data communication with each other. This is information indicating a posture (orientation).

  The storage device 101 is a generic name for devices having a function of storing information in the reference position acquisition device 10. In particular, the storage device 101 includes a program 102 executed by a CPU (not shown) of the reference position acquisition device 10, reference information 103 as individual information of the reference position acquisition device 10, and candidates acquired from the database server 11. Used to store information 112.

  As shown in FIG. 2, the database server 11 includes a storage device 110. In FIG. 2, the detailed configuration of the database server 11 is not shown, but the database server 11 includes a CPU, an operation unit, a display unit, a communication unit, and the like, and is configured as a general computer. ing.

  Unlike the reference position acquisition device 10, the database server 11 is a device that is installed without being limited to the vicinity of an area that provides augmented reality. As the installation location of the database server 11, for example, the center or backyard of the system operator is assumed. The database server 11 is connected to the reference position acquisition apparatus 10 via a network such as a LAN, the Internet, or a public network, and transmits candidate information 112 to the reference position acquisition apparatus 10 as necessary.

  The storage device 110 is a general term for devices having a function of storing information in the database server 11. In particular, the storage device 110 is used to store a program 111 executed by a CPU (not shown) of the database server 11 and candidate information 112.

  The candidate information 112 is information related to materials (contents) used when providing augmented reality, and is created by an operator, a designer, a programmer, or the like of the database server 11 and stored in the storage device 110. Specifically, the candidate information 112 is, for example, graphic information of a virtual object displayed in augmented reality, information about a position, information about time, or map information (layout data) in the augmented reality space 9. Note that each information included in the candidate information 112 has a tag (classification, description, etc.) that is referred to when selecting it as the output information 215.

  The candidate information 112 is generally different information for each augmented reality provided around the reference position acquisition device 10 and is transmitted from the database server 11 for each reference position acquisition device 10. Further, when changing the contents of the augmented reality being provided, the candidate information 112 is updated in the database server 11 and uploaded to the reference position acquisition apparatus 10.

  FIG. 3 is a diagram illustrating functional blocks included in the mobile terminal device 2 according to the first embodiment, together with a data flow. The card control unit 200, the position / orientation specifying unit 201, and the augmented reality configuration unit 202 illustrated in FIG. 3 are functional blocks realized by the CPU 20 operating according to the program 210.

  The card control unit 200 has a function of controlling the proximity wireless communication with the reference position acquisition device 10 by controlling the non-contact IC card unit 26. That is, the card control unit 200 constitutes an interface with the non-contact IC card unit 26, transfers the reference information 103 and the candidate information 112 received by the non-contact IC card unit 26 to the storage device 21, and stores them. Let In FIG. 3, it is not illustrated that some information is read from the storage device 21 and transmitted from the non-contact IC card unit 26, but such information may exist. That is, the non-contact IC card unit 26 may not be dedicated to writing.

  As described above, in the present embodiment, the close proximity wireless communication is started between the non-contact IC card unit 26 and the non-contact IC card reader unit 100 of the reference position acquisition device 10. It is determined that (sensor group 24) exists at the reference position. That is, the time when the card control unit 200 determines that the reference information 103 is received by the non-contact IC card unit 26 is when the mobile terminal device 2 (sensor group 24) is present at the reference position. Therefore, the card control unit 200 in the first embodiment corresponds to a determination unit according to the present invention.

  The position / orientation specifying unit 201 calculates a movement path as a result of relative positioning based on the measurement information 212 measured by the sensor group 24. Note that “information relating to movement” observed by the sensor group 24 includes information relating to rotational movement. Therefore, the movement path calculated by the position / orientation specifying unit 201 includes not only the position change history (movement locus) but also information related to the attitude change.

  Further, the position / orientation specifying unit 201 converts the position of the end point of the movement route to the absolute position based on the absolute position of the start point of the movement route obtained by the calculation, thereby allowing the mobile terminal device 2 (sensor group 24) to While specifying a present position, the present attitude | position of the portable terminal device 2 (sensor group 24) is specified. The absolute position of the starting point of the movement route is a reference position included in the reference information 103.

  That is, the position / orientation specifying unit 201 specifies the current position of the mobile terminal device 2 based on the reference information 103 stored in the storage device 21 and the measurement information 212 after receiving the reference information 103, It has a function of specifying the current attitude of the terminal device 2. In the present embodiment, “after receiving the reference information 103” is after the card control unit 200 determines that the sensor group 24 exists at the reference position. The measurement information 212 is information related to the movement measured by the sensor group 24. That is, the position / orientation specifying unit 201 in the first embodiment has a function corresponding to the position specifying unit and the attitude specifying unit according to the present invention.

  The current position and current posture of the mobile terminal device 2 identified by the position / orientation identifying unit 201 are stored in the storage device 21 as position information 214.

  The augmented reality configuration unit 202 refers to the position information 214 obtained by the position / orientation specifying unit 201 and the owner information 211 and extracts the output information 215 from the candidate information 112 that is a material for expressing augmented reality. It has a function.

  The owner information 211 is information related to the user that is input by the user when the operation unit 22 is operated. More specifically, the owner information 211 is information related to the characteristics of the object. Specifically, the age, gender, occupation, address, hobbies, preferences, behavior (purchasing) history, medical history (such as allergy) of the user who owns the mobile terminal device 2, marital status, family composition, property (car) Personal information such as home. These pieces of information are not limited to information directly input from the operation unit 22, and may be automatically collected by other applications.

  Further, in the present embodiment, the output information 215 is information displayed on the screen of the liquid crystal display in the display unit 23, and corresponds to information for extending reality in the augmented reality provided. The display unit 23 displays the augmented reality on the screen by displaying the output information 215 superimposed (synthesized) or added to the captured image information 213. The output information 215 may be processed by the augmented reality configuration unit 202 when extracted from the candidate information 112. That is, information regarding such a processing procedure may also be included in the candidate information 112.

  The above is the description of the configuration and functions of the augmented reality system 1 in the first embodiment. Next, a method for providing augmented reality to the user using the augmented reality system 1 will be specifically described.

  FIG. 4 is a flowchart showing an augmented reality providing method according to the first embodiment. In the present embodiment, an example will be described in which augmented reality is used to realize store guidance at a collective store such as a department store or a shopping mall. That is, an application that guides the user to a target store with the inside of the collective store as an augmented reality space 9 will be described as an example. Therefore, the candidate information 112 in the present embodiment includes a floor plan of the collective store, location information of each store arranged in the floor plan, advertisement information, coupon information, and the like.

  It is assumed that the portable terminal device 2 is activated, a predetermined initial setting is completed, and the owner information 211 is stored in the storage device 21 before each process shown in FIG. 4 is started. Further, it is assumed that the candidate information 112 is already stored in the storage device 101 of the reference position acquisition device 10. 4 shows each process for one user for convenience of explanation, the augmented reality system 1 provides augmented reality to a plurality of users (a plurality of portable terminal devices 2) at the same time. Is possible.

  When the user arrives at the collective store (augmented reality space 9) (step S1), the user performs communication enabling operation with respect to the reference position acquisition device 10 installed at the entrance using the portable terminal device 2 that he / she brought. (Step S2).

  In the first embodiment, if the user does not perform the communication enabling operation, it is difficult for the augmented reality system 1 to provide augmented reality to the user. Therefore, it is preferable to provide a mechanism for allowing the user to reliably perform the communication enabling operation when visiting the store.

  As such a mechanism, for example, it is conceivable that a store visit point is added to the mobile terminal device 2 by a communication enabling operation. If comprised in this way, the communication enabling operation | movement of the user who wants to accumulate a store visit point can be promoted more. Of course, a poster that prompts the customer (user) to enable communication may be posted near the entrance.

  Proximity wireless communication between the non-contact IC card unit 26 of the mobile terminal device 2 and the non-contact IC card reader unit 100 of the reference position acquisition device 10 by the communication enabling operation (step S2) performed by the user. Is started. Thereby, CPU20 (card | curd control part 200) of the portable terminal device 2 determines with Yes in step S3. That is, the card control unit 200 determines that the mobile terminal device 2 (sensor group 24) is present at the reference position when it determines Yes in step S3.

  When the wireless communication is started, the mobile terminal device 2 acquires the reference information 103 and the candidate information 112 from the reference position acquisition device 10 (step S4). As a result, the reference information 103 and the candidate information 112 are stored in the storage device 21.

  In parallel with the process of step S4, the imaging unit 25 starts imaging of the surroundings (in the augmented reality space 9) (step S5). Thereby, thereafter, the captured image information 213 is obtained according to the imaging cycle. In the present embodiment, the process of step S5 is automatically started by the communication enabling operation, but of course, it may be configured to start by a user instruction (user operation on the operation unit 22).

  In parallel with the process in step S5, the sensor group 24 starts measuring information related to movement (step S6). Thereby, thereafter, the measurement information 212 is updated according to the measurement cycle of the sensor group 24. That is, information regarding the movement of the user (mobile terminal device 2) in the augmented reality space 9 is continuously collected as the measurement information 212 by the sensor group 24.

  When Steps S4 to S6 are executed, the position / orientation specifying unit 201, based on the reference information 103 (information on the start point of the movement route) and the measurement information 212 (information for obtaining the movement route), the mobile terminal device 2 is identified (step S7), and position information 214 is created.

  Next, the augmented reality configuration unit 202 determines the absolute position and orientation in the augmented reality space 9 according to the position information 214, and determines the viewpoint and line-of-sight direction in the augmented reality. Further, the augmented reality configuration unit 202 extracts the output information 215 from the candidate information 112 according to the determined viewpoint and line-of-sight direction (step S8).

  If the viewpoint and the line-of-sight direction are determined, the visual field in the augmented reality space 9 can be determined. When the field of view in the augmented reality space 9 is determined, an object (virtual object) to be displayed virtually corresponding to the field of view and the shape of the object are determined. In this way, the augmented reality configuration unit 202 can select appropriate output information 215 from the candidate information 112. It should be noted that the conventional technique can be appropriately applied to the principle of selecting the output information 215 based on the position information 214 after the position information 214 is created.

  In the first embodiment, the output information 215 increases as the “viewpoint and line-of-sight direction” determined by the augmented reality configuration unit 202 matches the “imaging point (center point of the imaging range) and imaging direction” by the imaging unit 25. And the sensed discomfort when the captured image information 213 is combined and displayed are suppressed. Therefore, it is preferable that the position / orientation specifying unit 201 creates the position information 214 in consideration of this point (the position and orientation of the imaging unit 25 in the mobile terminal device 2).

  In store guidance at a collective store, even if the user's current position (position information 214) is the same, if the user's destination is different, it is necessary to change a virtual object (such as a guidance route) to be displayed. In other words, the output information 215 must be selected according to information different from the position information 214 such as the destination. Therefore, the augmented reality configuration unit 202 in the first embodiment extracts the output information 215 with reference to the owner information 211 as well as the position information 214.

  First, the augmented reality configuration unit 202 determines a store intended by the user from a plurality of stores in the collective store according to the owner information 211. As information for making such a determination, for example, a user's hobby, purchase history, store visit history, store search history, store name input as a destination, and the like included in the owner information 211 can be used. In general, information recorded as the owner information 211 is indefinite. Therefore, the augmented reality configuration unit 202 performs weighted (prioritized) in advance on highly probable information existing in the owner information 211, and performs evaluation by adding the weight of each information actually stored at the time of reference. Then, the user's target store is determined.

  In the case of an apparatus provided by a system operator, it is difficult to comprehensively collect owner information 211 that is personal information of the user. This is because a user who is worried about leakage of personal information is reluctant to provide his / her personal information to another person's device. However, since the mobile terminal device 2 in the first embodiment is owned by the user, it is predicted that the user's resistance to inputting personal information is small. Therefore, it is possible to collect the owner information 211 in advance accurately and in detail. It is also possible to prompt the user to input necessary information depending on the situation. Thereby, the augmented reality configuration unit 202 in the first embodiment can accurately predict the store desired by the user.

  If the store desired by the user can be predicted, the augmented reality configuration unit 202 outputs appropriate output information 215 from the candidate information 112 according to the field of view in the augmented reality space 9 determined based on the position information 214 and the predicted store name. Can be identified. Note that the augmented reality configuration unit 202 may determine a user's target store using public information such as time. For example, a method of preferentially selecting a restaurant at noon is also conceivable.

  When the output information 215 is created by executing step S8, the display unit 23 displays the output information 215 and the captured image information 213 in a synthesized manner on the screen of the liquid crystal display (step S9). Thereby, the display unit 23 expresses the augmented reality on the screen of the liquid crystal display, and provides the augmented reality to the user.

  Thereafter, while determining whether to end the provision of augmented reality (step S10), the processes of steps S7 to S10 are repeated until an instruction to end is given.

  FIG. 5 and FIG. 6 are diagrams showing a display example of the landscape of the augmented reality space 9 provided to the user in the first embodiment. That is, FIGS. 5 and 6 are examples of the augmented reality display screen displayed on the display unit 23.

  FIG. 5 shows a state in which an in-store image 213a, a route 215a that is a virtual object, and advertisements 215b, 215c, and 215d are displayed on the screen of the liquid crystal display.

  The in-store image 213a is captured image information 213 captured by the imaging unit 25, and is an image representing a real part in the augmented reality space 9.

  The route 215a and the advertisements 215b, 215c, and 215d are output information 215 selected from the candidate information 112, and are images that represent the augmented part (virtual part) in the augmented reality space 9.

  As illustrated in FIG. 5, the mobile terminal device 2 extends a landscape (route 215 a, advertisements 215 b, 215 c, and 215 d) composed of virtual objects over a real environment landscape (store image 213 a). A reality display screen can be created and provided.

  By visually recognizing the route 215a, the user can know that he / she is being guided toward the store D, and the route and distance to the store D can be easily compared with the case where the route is indicated on a map or the like. In addition, it is possible to know intuitively. Further, when the user is watching a map or the like, there is a risk of colliding with a passerby. However, in the augmented reality provided by the augmented reality system 1, a passerby is also displayed as the in-store image 213a, so that the user can easily perceive and avoid the danger of collision even when watching the screen.

  Further, the user can visually recognize the advertisements 215b, 215c, and 215d when confirming the route, and can also obtain fresh information on stores around the route. The advertisements 215b, 215c, and 215d can be easily adjusted to a position, angle, and size that are easy to see for the user facing the screen as compared to a pop advertisement or the like installed in front of an actual store, and have animation effects and the like. Since it can be easily provided, it is possible to effectively transmit information to the advertiser's store. In other words, it exhibits an excellent effect as an advertising medium for the store.

  On the other hand, the augmented reality configuration unit 202 can also prohibit the display of the advertisements 215b, 215c, and 215d of stores other than the store D by determining information related to stores that the user does not want from the owner information 211. Alternatively, in addition to this, an expression method of making a store portion other than the target store D invisible (for example, displaying a white wall image at an actual position of another store) is also possible. Thereby, user confusion due to unnecessary information being displayed can be suppressed, and an easy-to-use application suitable for the user can be provided.

  FIG. 6 shows a state in which an in-store image 215e, a virtual object route 215f, a star 215g, and coupons 215h and 215i are displayed on the screen of the liquid crystal display.

  The in-store image 215e is not a captured image information 213 imaged by the imaging unit 25 but a map image in which the actual store is deformed, and is an image representing a real part in the augmented reality space 9. In other words, the augmented reality configuration unit 202 can also create the in-store image 215e according to the layout and map of the augmented reality space 9 included in the candidate information 112. That is, the output information 215 is not necessarily limited to information representing only a virtual object.

  Further, the route 215f is output information 215 (schematic diagram) selected from the candidate information 112 based on the information calculated by the augmented reality configuration unit 202 based on the position information 214, the owner information 211, and the candidate information 112 (map information). It is information expressed in. The path 215f is an image representing an augmented part (virtual part) in the augmented reality space 9.

  Furthermore, the star mark 215g and the coupons 215h and 215i are output information 215 selected from the candidate information 112, and are images representing an augmented part (virtual part) in the augmented reality space 9.

  As illustrated in FIG. 6, the mobile terminal device 2 superimposes the landscape of the extended environment (path 215f, star 215g, coupons 215h, 215i) configured with virtual objects on the landscape of the real environment (in-store image 215e). The augmented reality display screen can be created and provided.

  The user can confirm the entire process up to the store D by viewing the route 215f. Further, by visually recognizing the star 215g, the current position in the collective store can also be confirmed.

  Further, the user can know that the coupon is issued at the stores C and D by visually checking the coupons 215h and 215i when confirming the route. Such a coupon is included in the candidate information 112, and is selected as output information 215 when facing a cash register of the corresponding store, and specific contents are displayed on the screen as a virtual object. That is, there is no need to present to the store clerk after the user operates and displays each one at the time of payment.

  Note that the mobile terminal device 2 can switch between the screen shown in FIG. 5 and the screen shown in FIG. 6 in accordance with a user instruction. However, the screen shown in FIG. 5 and the screen shown in FIG. 6 may be displayed side by side simultaneously. Further, when the mobile terminal device 2 detects that the user has arrived at the target store based on the position information 214, the mobile terminal device 2 may determine the next destination store and start guidance for the store. In addition, the start of guidance for the next destination store may be started when the user leaves the first store.

  As described above, the augmented reality system 1 according to the first embodiment includes the sensor group 24 that measures the measurement information 212, the storage device 21 (storage devices 101 and 110) that stores the reference position of the sensor group 24, and the like. The card control unit 200 that determines whether or not the sensor group 24 is present at the reference position, and the card control unit 200 stores the data in the storage device 21 after the sensor group 24 is determined to be present at the reference position. Based on the measured reference position and the measurement information 212 measured by the sensor group 24, the position / orientation specifying unit 201 for specifying the current position of the sensor group 24 and the sensor group 24 specified by the position / orientation specifying unit 201 And a display unit 23 for expressing augmented reality by outputting output information 215 according to the current position. As a result, the augmented reality system 1 can realize augmented reality without installing a marker or the like even in an environment where GPS signals cannot be received.

  Further, the storage device 21 stores the posture of the sensor at the reference position, and after the position / posture specifying unit 201 determines that the sensor group 24 exists at the reference position by the card control unit 200, the storage device 21 stores the posture. Based on the posture of the sensor group 24 at the reference position stored in the device 21 and the measurement information 212 measured by the sensor group 24, the current posture of the sensor group 24 is specified, and the display unit 23 specifies the position and posture. Output information 215 is output according to the current posture of the sensor group 24 specified by the unit 201. That is, unlike the GPS, the augmented reality system 1 can determine not only the absolute position of the user but also the posture and orientation of the user. And according to such information, an effective virtual object (output information 215) can be displayed at the tip of the user's line of sight. Therefore, this makes it possible to express more realistic augmented reality.

  In addition, the mobile terminal device 2 includes a reference position acquisition device 10 as an installation type device that is fixed at an absolute position, and the mobile terminal device 2 is in close proximity to the reference position acquisition device 10 when the sensor group 24 exists at the reference position. A non-contact IC card unit 26 that performs communication is provided. Thus, if the reference position acquisition device 10 is installed at a usable position immediately before the provision of augmented reality is started, the sensor group 24 can be reset immediately before providing the augmented reality. Therefore, error accumulation of the sensor group 24 over time can be suppressed.

  The reference position acquisition device 10 transmits the reference information 103 to the mobile terminal device 2 when close proximity wireless communication is performed between the mobile terminal device 2 and the reference position acquisition device 10. This eliminates the need for the mobile terminal device 2 to acquire the reference information 103 in advance.

  The reference position acquisition device 10 transmits candidate information 112 that is a candidate for the output information 215 to the mobile terminal device 2 when close proximity wireless communication is performed between the mobile terminal device 2 and the reference position acquisition device 10. . Thereby, it is not necessary for the mobile terminal device 2 to acquire the candidate information 112 in advance. Moreover, since the candidate information 112 is acquired immediately before use, the candidate information 112 in a relatively new state can be acquired.

  Further, the augmented reality system 1 stores owner information 211 as information related to the object to which the sensor group 24 is attached, and the display unit 23 outputs output information 215 corresponding to the stored owner information 211. Thereby, the augmented reality according to the target object can be provided. For example, when the object is an individual, augmented reality suitable for the individual can be provided for each individual.

  Note that the reference information 103 in the first embodiment has been described as being created in the reference position acquisition device 10 and stored in the storage device 101. However, for example, information corresponding to the reference information 103 may be created in the database server 11 and transmitted to each reference position acquisition apparatus 10 together with the candidate information 112.

  Further, the reference position acquisition device 10 and the database server 11 may be configured by a single computer.

  The candidate information 112 may be downloaded to the mobile terminal device 2 in advance by data communication between the communication unit 27 of the mobile terminal device 2 and the database server 11. That is, the candidate information 112 does not necessarily have to be acquired from the reference position acquisition device 10. In general, data communication by proximity wireless communication between the non-contact IC card unit 26 and the non-contact IC card reader unit 100 is not suitable for transmitting and receiving a huge amount of data. Therefore, it is preferable to transmit / receive candidate information 112 having a relatively large amount of data by data communication via a general network (such as the Internet). Before the user receives provision of augmented reality around the reference position acquisition device 10, the user accesses the database server 11 by operating the mobile terminal device 2 that he / she owns, and provides the augmentation provided around the reference position acquisition device 10. The actual candidate information 112 may be downloaded in advance. In this case, it is preferable to transmit the reference information 103 together with the candidate information 112 from the database server 11 to the mobile terminal device 2.

  In addition, in order for the augmented reality configuration unit 202 to predict the purchase behavior of the user and extract appropriate output information 215, information such as temperature and humidity indicating the surrounding environment may be effective. Therefore, environment sensors such as a temperature sensor and a humidity sensor may be provided in the mobile terminal device 2 so that the augmented reality configuration unit 202 refers to information collected by these sensors.

<2. Second Embodiment>
The mobile terminal device 2 in the first embodiment does not perform data communication for providing augmented reality in the augmented reality space 9. However, the present invention is not limited to such a form.

  FIG. 7 is a diagram illustrating an augmented reality system 1a according to the second embodiment. In addition, the number of the portable terminal device 2a and the terminal device 12 is not limited to the number shown by FIG. In addition, as a communication partner of the mobile terminal device 2a in the augmented reality space 9, at least one of the other mobile terminal device 2a and the terminal device 12 may exist.

  The augmented reality system 1a has the configuration in which the mobile terminal device 2a is provided instead of the mobile terminal device 2 and the stationary terminal device 12 is provided in the first embodiment. This is different from the configuration of the real system 1. Hereinafter, for the augmented reality system 1a according to the second embodiment, the same reference numerals are given to the same configurations as those of the augmented reality system 1 according to the first embodiment, and description thereof will be omitted as appropriate.

  FIG. 8 is a diagram illustrating functional blocks included in the mobile terminal device 2a according to the second embodiment, together with a data flow.

  The mobile terminal device 2a is a device having substantially the same configuration as the mobile terminal device 2 and moves in the augmented reality space 9 while being carried by the user. However, the communication unit 27 of the mobile terminal device 2a periodically searches for surrounding communication devices, and performs data by proximity wireless communication between the other mobile terminal device 2a and the terminal device 12 existing in the augmented reality space 9. Communicate. As the wireless communication method adopted here, for example, a proximity wireless communication method such as Bluetooth (registered trademark) is suitable. However, it is not limited to Bluetooth (registered trademark).

  The communication unit 27 of the mobile terminal device 2a has the owner information 211 stored in its own storage device 21 and the other mobile terminal device 2a and the terminal device 12 detected as communication devices in the augmented reality space 9 and The position information 214 is transmitted. The owner information 211 transmitted to the outside by the communication unit 27 is limited to information permitted by the user in order to prevent leakage of personal information.

  Further, the communication unit 27 of the mobile terminal device 2a stores information received from the other mobile terminal device 2a or the terminal device 12 as candidate information 112 in the storage device 21 of the own device. In other words, in the second embodiment, the candidate information 112 is not limited to the information acquired from the reference position acquisition device 10, but information collected from other mobile terminal devices 2a and terminal devices 12 is used. included.

  The terminal device 12 is a general stationary computer, and is a device whose absolute position is fixed in the augmented reality space 9. The candidate information 112 in the second embodiment includes identification information of the terminal device 12 and information on the absolute position (arrangement position). Further, the terminal device 12 has a function of performing data communication by proximity wireless communication with the communication unit 27 of the mobile terminal device 2a, and information (details will be described later) unique to the own device is stored in the mobile terminal device 2a. Send to.

  Hereinafter, the augmented reality system 1a according to the second embodiment will be described with an example of an application in which a game center where a large number of game machines (terminal devices 12) are installed is an augmented reality space 9. However, for easy understanding of the explanation, an example of augmented reality realized by the mobile terminal device 2a performing wireless communication with another mobile terminal device 2a, and the mobile terminal device 2a performing wireless communication with the terminal device 12 are described. A description will be given separately from an example of augmented reality realized by doing.

  First, an example of augmented reality that is realized by the mobile terminal device 2a performing wireless communication with another mobile terminal device 2a in the augmented reality system 1a will be described.

  FIG. 9 is a diagram illustrating an example of augmented reality realized by wireless communication between the mobile terminal devices 2a. In the example shown in FIG. 9, an in-game center image 213b, an avatar image 215j, and a message 215k are displayed on the display unit 23 of the mobile terminal device 2a.

  The game center image 213b is a video (captured image information 213) in the game center (augmented reality space 9) captured by the imaging unit 25 of the mobile terminal device 2a. That is, the in-game center image 213b is an image representing a real part in augmented reality. Here, three terminal apparatuses 12 are imaged.

  The avatar image 215j and the message 215k are images in which the output information 215 selected from the candidate information 112 by the augmented reality configuration unit 202 is displayed. That is, the avatar image 215j and the message 215k are images that represent virtual things that do not exist in reality, and are images that represent an extended portion in augmented reality.

  The avatar image 215j and the message 215k are both information selected from the candidate information 112, but are not information acquired from the reference position acquisition device 10. These are information created based on the owner information 211 and the position information 214 received from the other mobile terminal device 2a.

  In the second embodiment, the user acquires the reference information 103 and the candidate information 112 from the reference position acquisition device 10 at the entrance of the augmented reality space 9 as in the first embodiment, and enters the game center. enter. Further, the user edits the owner information 211 at an arbitrary timing (that is, inside and outside the game center), and is provided by the user's avatar and various messages and a game (terminal device 12) installed in the game center. ) Play history or self profile.

  In the game center, the communication unit 27 searches for a nearby communication device (another portable terminal device 2a) and starts communication with the detected other portable terminal device 2a. The portable terminal device 2a exchanges the owner information 211 and the position information 214 with each other with the other portable terminal device 2a. Then, candidate information 112 is created based on the received owner information 211 and position information 214 of the other portable terminal device 2a, and stored in the storage device 21 of the own device.

  The augmented reality configuration unit 202 selects the output information 215 from the candidate information 112 when the field of view in the augmented reality space 9 is obtained as in the first embodiment. At this time, if another mobile terminal device 2a exists in the field of view, the output information 215 is selected from the candidate information 112 created based on the owner information 211 received from the other mobile terminal device 2a. Note that the current position of the other mobile terminal device 2a can be determined from the position information 214 received from the other mobile terminal device 2a (more specifically, the candidate information 112 derived from the position information 214).

  In this way, the mobile terminal device 2a converts the avatar (avatar image 215j) set in the owner information 211 by the user of the other mobile terminal device 2a into the current position of the user and the actual image of the user. Overwrite and display. In addition, a message (message 215k) set in the owner information 211 received from the other mobile terminal device 2a can be displayed.

  As described above, the augmented reality system 1a according to the second embodiment exchanges the owner information 211 and the position information 214 with each other between the plurality of portable terminal devices 2a. As a result, users visiting the game center can send messages, pictograms, characters (avatars) between each user, introductions of each user using the game play history (fighting game masters, music game beginners, etc.), etc. Can interact and display and enjoy.

  Next, an example of augmented reality that is realized by the mobile terminal device 2a performing wireless communication with the terminal device 12 in the augmented reality system 1a will be described.

  FIG. 10 is a diagram illustrating an example of augmented reality realized by wireless communication between the mobile terminal device 2 a and the terminal device 12. In the example shown in FIG. 10, an in-game center image 213c, character images 215m and 215n, and a message 215p are displayed on the display unit 23 of the mobile terminal device 2a.

  The game center image 213c is a video (captured image information 213) in the game center (augmented reality space 9) captured by the imaging unit 25 of the mobile terminal device 2a. That is, the in-game center image 213c is an image that represents a real part in augmented reality. Here, four terminal apparatuses 12 are imaged. In the example illustrated in FIG. 10, in order to distinguish each terminal device 12, alphabets are added to the respective symbols to form terminal devices 12 a, 12 b, 12 c, and 12 d.

  The character images 215m and 215n and the message 215p are images that display the output information 215 selected from the candidate information 112 by the augmented reality configuration unit 202. That is, the character images 215m and 215n and the message 215p are images that represent virtual things that do not exist in reality, and are images that represent an extended portion in augmented reality.

  The character images 215m and 215n and the message 215p are all information selected from the candidate information 112, but are not information acquired from the reference position acquisition device 10. These are information created based on information unique to each terminal device 12a, 12b, 12c received by the mobile terminal device 2a.

  As already described, in the second embodiment, the communication unit 27 searches for nearby communication devices in the game center. At this time, when the terminal device 12 is detected, the communication unit 27 starts communication with the detected terminal device 12. The mobile terminal device 2a receives information unique to the terminal device 12 from the terminal device 12 that has started communication. Then, candidate information 112 is created based on the received unique information and stored in the storage device 21 of the own device.

  Note that the position of the terminal device 12 is included in the candidate information 112 acquired from the reference position acquisition device 10. Therefore, the mobile terminal device 2a does not receive unique information from all the terminal devices 12 for which close proximity wireless communication is established, but is present in the user's field of view based on the position of the terminal device 12 acquired in advance. The unique information may be received only from the determined terminal device 12. Thereby, the information amount transmitted / received in data communication can be suppressed.

  Similarly to the mobile terminal device 2 in the first embodiment, the mobile terminal device 2a can also determine the user's visual field in the augmented reality space 9 by determining the user's viewpoint and line-of-sight direction. Therefore, the augmented reality configuration unit 202 includes candidate information 112 (candidate information 112 created based on unique information received from the terminal device 12) derived from the terminal device 12 existing in the user's field of view. Output information 215 can be selected.

  In this way, the mobile terminal device 2a displays the unique information of each terminal device 12 at a position corresponding to the position of each terminal device 12. In the example shown in FIG. 10, a character image 215m corresponding to the play status of the terminal device 12a, a character image 215n corresponding to the play status of the terminal device 12b, and a message 215p indicating the reception status of the terminal device 12c are displayed. Yes.

  As described above, the augmented reality system 1a according to the second embodiment collects information unique to the terminal device 12 existing in the augmented reality space 9 in the mobile terminal device 2a. Thereby, the user who visited the game center can receive information such as play information, demo information, and how to play provided games from the nearby terminal device 12 and display them as virtual objects. In addition to being close, it is possible to determine the visual field of the user in the augmented reality space 9 and display the virtual object at the end of the user's line of sight. Therefore, a real augmented reality can be expressed, and an augmented reality can be expressed based on more real-time information than the first embodiment. Also, by expressing the play situation as augmented reality, the play scenery (character images 215m, 215n) is displayed, and the game can be enjoyed as a spectator.

  Information that is not frequently changed, such as demo information and how to play, may be received from the reference position acquisition device 10 as candidate information 112. That is, the output information 215 is not limited to information received from the other portable terminal device 2a or the terminal device 12.

<3. Third Embodiment>
In the above embodiment, the information collected regarding the object is only the owner information 211 and the position information 214, but is not limited to these information. Further, in the above-described embodiment, an example has been described in which the real part is displayed on the display unit 23 as image information in the provided augmented reality. However, the real part in augmented reality does not necessarily have to be displayed as image information.

  FIG. 11 is a diagram illustrating an augmented reality system 1b according to the third embodiment. The augmented reality system 1b includes the mobile terminal device 2b instead of the mobile terminal device 2, and the point that the configuration corresponding to the reference position acquisition device 10 and the database server 11 is not provided. It differs from the augmented reality system 1 in the form. Hereinafter, for the augmented reality system 1b according to the third embodiment, the same reference numerals are given to the same configurations as those of the augmented reality system 1 according to the first embodiment, and description thereof will be omitted as appropriate.

  As shown in FIG. 11, the portable terminal device 2b is configured as an HMD (Head Mounted Display) type device, and moves along with the user when the user wears the head. As shown in the first embodiment, when the handheld portable terminal device 2 is adopted, the relative positional relationship between the user and the portable terminal device 2 changes depending on the state of the user's handheld, This causes an error between the viewpoint in the augmented reality obtained from the position of the sensor group 24 (position information 214) and the imaging point of the imaging unit 25. However, the augmented reality system 1b according to the present embodiment employs the wearable portable terminal device 2b, so that the reality portion and the augmented portion are visually compared with the case where the handheld portable terminal device 2 is employed. The accuracy of matching is improved. Therefore, the reality of the provided augmented reality is improved.

  FIG. 12 is a block diagram of the mobile terminal device 2b in the third embodiment.

  The mobile terminal device 2b is usually a dedicated device owned by the system operator. Therefore, as shown in FIG. 12, the storage device 21 does not store information corresponding to the owner information 211 regarding the user.

  The portable terminal device 2b includes a display unit 23a having a transmissive display. Real things arranged in the augmented reality space 9 are perceived and perceived by the user's vision based on the light transmitted through the display. Therefore, the augmented reality system 1b according to the third embodiment does not display the real part image information when providing augmented reality. On the other hand, the display unit 23a displays the output information 215 at a predetermined position on the display, thereby appropriately superimposing the virtual object (extended portion) on the real part.

  Further, the mobile terminal device 2b does not include the imaging unit 25 and does not have a function of imaging the surroundings. Therefore, in the third embodiment, information corresponding to the captured image information 213 is not created. This is because, as already described, in the mobile terminal device 2b, it is not necessary to display the real part on the screen.

  Furthermore, the mobile terminal device 2b does not include a configuration corresponding to the non-contact IC card unit 26 and the communication unit 27, and is configured as a stand-alone device. In the storage device 21 of the portable terminal device 2b, the reference information 103 and the candidate information 112 are stored together with the program 210 in advance.

  The portable terminal device 2b includes not only the sensor group 24 but also a biological sensor 28. The biological sensor 28 is a device having a function of measuring biological information 216 related to a living body. As the biological sensor 28, for example, a heart rate sensor that measures the heart rate of the user, a respiration sensor that measures information related to respiration such as the user's respiration rate, a microphone that measures the sound generated by the user, and the like are assumed. However, the biological sensor 28 is not limited to such a device, and may be any device having a function of collecting information that can be used to determine the current physiological state of the user.

  The mobile terminal device 2b includes a speaker 29 that reproduces sound based on information related to sound. In particular, the speaker 29 is used as an output unit that outputs information related to sound included in the output information 215 as sound.

  FIG. 13 is a diagram illustrating functional blocks included in the mobile terminal device 2b according to the third embodiment, together with a data flow. The mobile terminal device 2b does not include the card control unit 200, but includes a position / orientation specifying unit 201a and an augmented reality configuring unit 202a instead of the position / orientation specifying unit 201 and the augmented reality configuring unit 202. Different from the device 2.

  The position / orientation specifying unit 201a determines that the mobile terminal device 2b (sensor group 24) is present at the reference position and the current posture is the posture at the reference position in accordance with the input information from the operation unit 22. To do. In the present embodiment, the current position and current posture of the mobile terminal device 2 b when the reset button of the operation unit 22 is operated are reset by the reference information 103. That is, in the third embodiment, the position / orientation specifying unit 201a has a function corresponding to the determination unit according to the present invention.

  The augmented reality configuration unit 202a extracts the output information 215 from the candidate information 112 according to the position information 214, as in the first embodiment. However, in the third embodiment, since there is no information corresponding to the owner information 211, the augmented reality configuration unit 202a does not refer to the owner information 211 when extracting the output information 215. Instead, the augmented reality configuration unit 202 extracts the output information 215 according to the biological information 216. Thereby, the augmented reality system 1b (the display unit 23a and the speaker 29) in the third embodiment outputs output information 215 corresponding to the biological information 216 measured by the biological sensor 28.

  Hereinafter, an augmented reality system 1b according to the third embodiment will be described using an example of an application in which a haunted house is an augmented reality space 9.

  FIG. 14 is a flowchart showing an augmented reality providing method according to the third embodiment.

  First, the person in charge of the haunted house operates the reset button (operation unit 22) (step S11) with the portable terminal device 2b stationary at a predetermined position in a predetermined posture (step S11), and the portable terminal device 2b. In a state that can be delivered to the user (hereinafter referred to as “standby state”).

  The predetermined position is a position that matches the reference position stored in the reference information 103. Further, the predetermined posture is a posture stored in the reference information 103 (a posture defined as a posture at the reference position). That is, in the third embodiment, the person in charge of the window executes an operation corresponding to the communication enabling operation in the first embodiment.

  By executing step S11, the sensor group 24 starts measuring the measurement information 212 (step S12), and the position / orientation specifying unit 201a starts creating the position information 214 based on the reference information 103 and the measurement information 212. (Step S13).

  Next, when the user arrives at the entrance of the haunted house, the person in charge of the window hands over the portable terminal device 2b in the standby state to the user. Then, the user wears the received mobile terminal device 2b (step S14). Accordingly, the display unit 23a is disposed in front of the user's eyes, the speaker 29 is disposed in the vicinity of the user's ear, and the biosensor 28 is attached to the user's body.

  The user enters the haunted house (augmented reality space 9) while wearing the mobile terminal device 2b (step S15).

  The augmented reality configuration unit 202a determines whether or not the user (mobile terminal device 2b) exists in the augmented reality space 9 according to the position information 214 (step S16). It is determined whether or not it is ON (step S17). The flag is information indicating whether or not the user has entered the augmented reality space 9, and is set to “ON” when the user has entered, and “OFF” when the user has not entered. Set to

  In the case of No in step S17, it indicates that the user has not yet entered the augmented reality space 9, so that it is considered that the user's entry action has not yet been completed, and the process returns to step S16.

  On the other hand, when the user exists in the augmented reality space 9 (Yes in step S16), the CPU 20 sets the flag to ON (step S18). Then, the augmented reality configuration unit 202a creates output information 215 based on the position information 214 and the biological information 216 (step S19).

  When step S19 is executed, the display unit 23a and the speaker 29 output the output information 215 (step S20), thereby expressing augmented reality. The processes in steps S16 to S20 are continued until it is determined that the user does not exist in the augmented reality space 9 (exited from the exit).

  If it is determined Yes in step S17, it is assumed that the user who has entered the augmented reality space 9 has left the augmented reality space 9, and the CPU 20 sets the flag to OFF (step S21), and the portable terminal device 2b. The provision of augmented reality by will end. Thereafter, the contact person collects the portable terminal device 2b from the user.

  Next, how the biological information 216 is used for selecting the output information 215 in the present embodiment will be described.

  FIG. 15 is a diagram illustrating an example of augmented reality realized by the augmented reality system 1b according to the third embodiment. The willow image 215q and the ghost image 215r shown in FIG. 15 are both output information 215. On the other hand, things other than the willow image 215q and the ghost image 215r exist in reality that are perceived by the light transmitted through the transmissive display.

  FIG. 16 is a diagram illustrating the display position of the ghost image 215 r in the augmented reality space 9. In the example shown in FIG. 16, eight places for displaying the ghost image 215 r in the augmented reality space 9 are set. In other words, in the candidate information 112, ghost images 215r to be displayed at eight locations are prepared in advance. A circle 90 shown in FIG. 16 indicates a determination position (described later). Further, the shaded portions shown in FIG. 16 indicate actual walls and pillars.

  17 to 19 are diagrams showing display examples of the ghost image 215r. In FIG. 17 to FIG. 19, arrows indicated by bold lines indicate the user's route in the augmented reality space 9.

  The augmented reality system 1b according to the third embodiment changes the position at which the ghost image 215r is actually displayed according to the physiological state of the user when the user reaches the determination position (circle 90). That is, based on the biological information 216 at the determination position, the ghost image 215r is displayed only at the position shown in FIG. 17 when the heart rate is larger than 120 [bpm], and when the heart rate is 90 to 120 [bpm] in FIG. The ghost image 215r is displayed only at the position shown. When the heart rate is less than 90 [bpm], the ghost image 215r is displayed only at the position shown in FIG.

  As described above, for the user who is determined to be greatly surprised by analyzing the biological information 216, the ghost image 215r is displayed so as to be a relatively short distance and a simple movement route (FIG. 17). Further, the number of ghosts (ghost images 215r) encountered is the smallest.

  On the other hand, the ghost image 215r is displayed so as to be a relatively long-distance and complicated movement route (FIGS. 18 and 19) for a user who is determined not to be surprised very much. Also, the number of ghosts encountered (ghost images 215r) is set to increase.

  As described above, the augmented reality system 1b according to the third embodiment outputs the augmented reality according to the physiological state of the living body by outputting the output information 215 corresponding to the living body information 216 measured by the living body sensor 28. Can be provided.

  Note that the extent to which the user is surprised can also be determined by, for example, using an acceleration sensor and counting the number of times the acceleration has changed significantly. For example, the display pattern of FIG. 17 is applied to a user who has been surprised twice or more between the time of entry and the determination position, the display pattern of FIG. A method of applying the display pattern of FIG. 19 is conceivable.

  FIG. 20 is a diagram illustrating a modification in the case where the display of the ghost image 215r is changed in the third embodiment. A broken-line arrow in FIG. 20 indicates a locus when the display position of the ghost image 215r is sequentially changed.

  17 to 19, it is determined whether or not to display each ghost image 215 r based on the biological information 216. However, for a user who can hardly be surprised, for example, as shown in FIG. 20, even if the display position of a specific ghost image 215r is sequentially changed as if the ghost image 215r tracks the user. Good.

  In this embodiment, the user's route is changed by changing the display of “ghost” as a virtual object, but the method of changing the route is not limited to this. For example, by displaying a wall as a virtual thing between the walls that are real things, it is possible to change the route so that the walls are connected so that they cannot pass.

  In the augmented reality system 1b according to the third embodiment, similarly to the augmented reality system 1 according to the first embodiment, devices corresponding to the reference position acquisition device 10 and the database server 11 are provided, and a mobile terminal device is provided. You may provide the non-contact IC card part 26 and the communication part 27 in 2b. With this configuration, the augmented reality system 1b according to the third embodiment can easily cope with the update of the reference information 103 and the candidate information 112.

  In particular, in the application imitating the “haunted house” shown in the third embodiment, it is preferable to change the actual layout, the display form of the virtual object, and the like in a relatively short cycle in order to call the repeater. In that case, it is assumed that the reference information 103 and the candidate information 112 need to be updated. However, even in such a case, the updated reference information 103 and candidate information 112 can be stored in a portable recording medium such as an SD card and supplied to the portable terminal device 2b. is there.

<4. Fourth Embodiment>
The application to which the present invention is applied is not limited to that shown in the above embodiment, and many more variations are conceivable.

  FIGS. 21 and 22 are augmented reality display examples of the search application provided by the augmented reality system 1c according to the fourth embodiment. The search application is for the user to search for an object (virtual object) such as a treasure box installed in the augmented reality space 9 using the mobile terminal device 2. The augmented reality system 1c in the fourth embodiment can be realized, for example, with the same configuration as the augmented reality system 1 in the first embodiment.

  As shown in FIG. 21, the display unit 23 combines the captured image information 213d of the real part, the treasure box image 215s as the target object, the message 215t indicating the information for searching and the compass image 215u. Is displayed. Here, the treasure box image 215 s, the message 215 t, and the compass image 215 u are output information 215 selected from the candidate information 112.

  In the augmented reality space 9 provided by the search application, the user searches for a virtual object (treasure box) using the message 215t and the compass image 215u output to the mobile terminal device 2.

  Then, when the current position and current posture of the user are within a predetermined range, the object is found. That is, on the condition that the position information 214 falls within a predetermined range, the augmented reality configuration unit 202 selects the treasure box image 215s as the output information 215, and a screen as shown in FIG. 22 is displayed. In the example shown in FIG. 22, a treasure chest image 215 s and a message 215 v indicating that a treasure chest has been found are displayed along with a captured image 213 e that represents a real part.

  In the fourth embodiment, an example in which a treasure chest is searched as an object has been described. However, the object is not limited to this. For example, the target object can be an animal expressed as a virtual object. In that case, it is preferable to adjust the cry of the animal according to the position information 214 of the user and output it from the speaker 29 as information for clues. The sound adjustment such as squeal is not only the volume adjustment according to the distance between the animal as the target object and the user, but also the sound source direction that the user knows by mutually adjusting the volume reaching the left and right ears of the user It is also effective to adjust.

<5. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made.

  For example, each process shown in the above embodiment is merely an example, and is not limited to the order and contents shown above. That is, as long as the same effect can be obtained, the order and contents may be changed as appropriate. For example, the step (step S5) in which the imaging unit 25 starts imaging and the step (step S6) in which the sensor group 24 starts measurement may be interchanged.

  In addition, the functional blocks (the card control unit 200, the position / orientation specifying unit 201, the augmented reality configuration unit 202, and the like) described in the above embodiment are described as being realized by software as the CPU 20 operates according to the program 210. did. However, some or all of these functional blocks may be configured by a dedicated logic circuit and realized in hardware.

  Further, the index means may be a bar code expressing information on the reference position and the posture at the reference position. For example, a barcode that is read in a specific posture at the reference position may be provided in the vicinity of the augmented reality space, and this may be configured to be imaged and read by the imaging unit 25.

  In the above-described embodiment, the example in which the sensor group 24 and the output unit (the display units 23 and 23a and the speaker 29) are provided in the same device has been described. However, they may be present in another device. For example, by attaching a sensor group 24 to a pet (target object), releasing it in the augmented reality space 9, and displaying a virtual object on an output means provided in a device carried by the user according to the movement of the pet. Augmented reality may be provided. Alternatively, a ball (object) containing the sensor group 24 is thrown by the user in the augmented reality space 9, and the trajectory of the ball is calculated according to the position and acceleration at the moment of being thrown, and a virtual corresponding to the ball is calculated. An application that displays the trajectory of an object (for example, a fireball by a spear or magic) or the state of an enemy who is a target on an apparatus (output means) at hand of the user is also assumed.

1, 1a, 1b, 1c Augmented reality system 10 Reference position acquisition device 100 Non-contact IC card reader unit 21, 101, 110 Storage device 102, 111, 210 Program 103 Reference information 11 Database server 112 Candidate information 12, 12a, 12b, 12c, 12d terminal device 2, 2a, 2b portable terminal device 20 CPU
200 Card control unit 201, 201a Position / attitude specifying unit 202, 202a Augmented reality configuration unit 211 Owner information 212 Measurement information 213, 213d Captured image information 213a Store image 213b, 213c Game center image 214 Position information 215 Output information 215a Route 215b , 215c, 215d Advertisement 215e Store image 215f Path 215g Star sign 215h, 215i Coupon 215j Avatar image 215k, 215p, 215t, 215v Message 215m, 215n Character image 215q Willow image 215r Ghost image 215s Treasure box image 215u Operation unit 23, 23a Display unit 24 Sensor group 25 Imaging unit 26 Non-contact IC card unit 27 Communication unit 28 Biosensor 29 Speaker 9 Augmented reality space 90 yen

Claims (6)

A mobile terminal device whose position is variable;
An indicator means whose absolute position is known;
With
A sensor that measures information about movement;
First storage means for storing a reference position of the sensor and an attitude of the sensor at the reference position;
Obtaining means for obtaining individual information of the indicator means;
Determination means for determining whether or not the sensor is present at the reference position;
After the determination means determines that the sensor is present at the reference position, based on the reference position stored in the first storage means and information on the movement measured by the sensor. Position specifying means for specifying the current position of the sensor;
After the determination means determines that the sensor is present at the reference position, the attitude of the sensor at the reference position stored in the first storage means and the movement measured by the sensor Posture identifying means for identifying the current posture of the sensor based on the information;
Output means for expressing augmented reality by outputting output information according to the current position of the sensor specified by the position specifying means and the current attitude of the sensor specified by the attitude specifying means;
Further comprising
The indicator means includes a housing that defines an attitude of the sensor when the sensor is present at the reference position so as to be an attitude of the sensor at the reference position stored in the first storage means. Prepared,
The sensor and the acquisition unit are provided in the mobile terminal device,
The augmented reality system in which the determination unit determines that the sensor is present at the reference position when the acquisition unit acquires the individual information of the index unit. The augmented reality system according to claim 1,
A plurality of the mobile terminal devices;
The portable terminal device
A second communication means for performing wireless data communication with another portable terminal device existing in the augmented reality space;
The output means;
With
The second communication means receives the current position of the sensor included in the other mobile terminal device by wireless data communication,
The augmented reality system in which the output means outputs output information according to a current position of the sensor included in the other portable terminal device received by the second communication means. The augmented reality system according to claim 1,
The portable terminal device
Third communication means for performing wireless data communication with a terminal device existing in the augmented reality space;
The output means;
With
The third communication means receives specific information about the terminal device by wireless data communication,
The augmented reality system in which the output means outputs output information according to specific information about the terminal device received by the third communication means. An augmented reality system according to any one of claims 1 to 3,
Further comprising second storage means for storing information relating to the object associated with the sensor;
The augmented reality system in which the output means outputs output information corresponding to information on the object stored in the second storage means. An augmented reality system according to any one of claims 1 to 4,
A biological sensor for measuring biological information about the living body;
The output means is an augmented reality system that outputs output information corresponding to biological information measured by the biological sensor. An augmented reality providing method using a mobile terminal device whose position is variable,
Measuring information about movement with a sensor;
Storing a reference position of the sensor and a posture of the sensor at the reference position in a first storage unit;
Obtaining the individual information of the indicator means whose absolute position existing outside is known;
The orientation of the sensor when the sensor is present at the reference position is defined by a housing provided in the indicator means so as to be the orientation of the sensor at the reference position stored in the first storage means. Process,
Determining whether the sensor is present at the reference position;
After it is determined that the sensor is present at the reference position, based on the reference position stored in the first storage unit and information on the movement measured by the sensor, the sensor A process of identifying the current position;
After determining that the sensor is present at the reference position, based on the attitude of the sensor at the reference position stored in the first storage unit and information on the movement measured by the sensor. Identifying the current posture of the sensor;
Expressing augmented reality by outputting output information according to the current position of the identified sensor and the current posture of the identified sensor;
Have
An augmented reality providing method for determining that the sensor is present at the reference position in the determining step when the individual information of the index means is acquired .

Images