JP2018128815A - Information presentation system, information presentation method and information presentation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a novel way of showing and a novel way of enjoying by fusing the augmented reality and virtual reality that have been discretely and independently handled up to now.SOLUTION: A system comprises a camera part 121 imaging an image in a real space, a position information acquisition part 147 acquiring a position and direction in a coordinate system in the real space as current position information, an information acquisition part 141 acquiring real map information including geographic information in the coordinate system in the real space and virtual map information where a virtual object is arranged on geographic information in a coordinate system in a virtual space, a display data generation part 142 generating augmented display data by superimposing the virtual object on the image that the camera part imaged on the real map information as well as generating virtual display data where the virtual object is arranged on the virtual map information, and a display control part 143 implementing display transition processing that selectively switches the virtual display data with the augmented display data or synthesizing a part or whole of the other display data on one display data.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an information presentation system, an information presentation method, and an information presentation program for presenting information related to a virtual object arranged in a virtual coordinate system of a virtual space.

  In recent years, in the field of information processing for displaying content, virtual reality (VR) that constitutes a virtual space by computer graphics, augmented reality (AR) in which video in the real space is processed by a computer and information is added. ; Augmented Reality) is drawing attention. Application of VR technology is expected to be applied to more and more fields such as manufacturing, medical treatment, games, daily life, etc., using a head mounted display to change display information according to the movement of the user. . On the other hand, as an application of the AR technology, there has been proposed a viewing mirror that superimposes and displays additional information such as tourist information on a real-space landscape image captured by a camera (for example, Patent Document 1). In the observing mirror disclosed in Patent Document 1, an actual local landscape and a virtual image of a ruin or the like are conceived alternately in the head by synthesizing and displaying a corresponding virtual image with a photographic image to be viewed. Without compulsory work, it is possible to provide a user with a composite image that is accurate in the scale of a building or the like and in a positional relationship on the terrain and rich in reality.

JP 2003-264740 A

  However, in the conventional content display methods, the VR technology and the AR technology are used as systems and services for different uses and purposes, so that independent drawing is performed in each coordinate system, and each use and purpose is displayed. Projected video based on. For this reason, augmented reality (AR) and virtual reality (VR) are used as separate separated contents, resulting in discontinuity between real space and virtual space, and there is a certain limit to its expressive power. there were.

  Therefore, the present invention solves the above-described problems, and in the content provided by information processing, the augmented reality and the virtual reality that have been conventionally separated and treated independently are merged. Thus, an object is to provide an information presentation system, an information presentation method, and an information presentation program that can realize a new way of viewing and enjoying.

In order to solve the above problems, the present invention is a system for presenting information on a virtual object placed in a virtual coordinate system of a virtual space,
A camera unit that captures images of real space;
A position information acquisition unit that acquires the position and direction in the coordinate system of the real space as current position information;
A real map acquisition unit for acquiring real map information including geographical information in the coordinate system of the real space;
A virtual map acquisition unit that acquires virtual map information in which the virtual object is arranged on geographical information in the coordinate system of the virtual space;
A virtual reality display data generation unit that generates virtual display data in which the virtual object is arranged on the virtual map information shared with the coordinate system in the real space;
Augmented reality display data for generating augmented display data by superimposing display data in which the virtual object is arranged on the real map information associated with a coordinate system in the real space on the video imaged by the camera unit A generator,
According to the current position information acquired by the position information acquisition unit, the virtual display data and the extended display data are selectively switched, or a part or all of the other display data is switched to one display data. A display control unit that executes display transition processing to be combined;
And a display unit that displays the virtual display data and the extended display data that have undergone display transition processing by the display control unit.

Further, the present invention is a method for presenting information about a virtual object arranged in a virtual coordinate system of a virtual space,
A real space information acquisition step in which the camera unit captures an image of real space, and the position information acquisition unit acquires the position and direction in the coordinate system of the real space as current position information;
Real map information including geographic information in the coordinate system of the real space and virtual map information in which the virtual object is arranged on the geographic information in the coordinate system of the virtual space are acquired, and a virtual display data generation unit is configured to acquire the real space. And generating virtual display data in which the virtual object is arranged on the virtual map information shared with the coordinate system in the map, and an extended display data generation unit is associated with the coordinate system in the real space. A display data generation step of generating extended display data by superimposing display data in which the virtual object is arranged on the information on the video captured by the camera unit;
According to the current position information acquired by the position information acquisition unit, the virtual display data and the extended display data are selectively switched, or a part or all of the other display data is switched to one display data. A display control step in which the display control unit executes display transition processing to be combined; and
A display step in which the display control unit displays the virtual display data and the extended display data subjected to the display transition process in the display control step;
It is characterized by including.

  According to the present invention as described above, when displaying augmented reality (AR) and virtual reality (VR), the coordinate systems are integrated to display augmented reality (AR) and virtual reality (VR) as one coordinate. It can be mixed in the system. As a result, by combining AR, it is possible to produce a virtual experience that is rich in diversity and highly realistic.

  In the above invention, a transition setting unit that sets a boundary line or boundary surface for executing the display transition process on the real map information or the virtual map information, and the position information included in the current position information includes the boundary It is preferable to further include a trigger determination unit that determines that the line or the boundary surface has been passed, and the display control unit performs the display transition process based on a determination result by the trigger determination unit.

  In this case, the augmented reality and the virtual reality are automatically switched without requiring any special operation by passing through a boundary line or boundary surface set on the real map information or the virtual map information. And can make a natural transition from augmented reality to virtual reality or from virtual reality to augmented reality.

  In the above-described invention, the display control unit further includes a scenario management unit that acquires and manages scenario data describing a time at which the display transition process is executed, a place, types of objects to be presented, and their appearance order, and the display control unit includes: It is preferable to execute the display transition process according to the scenario data or a user operation based on the determination result by the trigger determination unit.

  In this case, augmented reality and virtuality are automatically or according to scenario data, or any operation by the user, or user operation according to the scenario data, with the time, place, types of objects to be presented, and their order and timing. It is possible to switch between reality and improve the diversity of expressive power and to make various effects. In the above invention, it further includes a main body unit fixed to the coordinate system of the real space and installed so as to be able to turn left and right and to be lifted up and down, and the camera unit is disposed at a tip of the main body unit, and the display The part is preferably disposed at the rear end of the main body. In this case, the present invention can be realized, for example, in the form of a viewing mirror installed in a sightseeing spot, and a virtual or expansive effect can be added to the scenery seen by the viewing mirror.

  In the above invention, the object is preferably a plate-shaped signboard having an opening. In this case, for example, an object such as an “appearance signboard” installed in a sightseeing spot can be arranged virtually or in an expanded manner.

  The information presentation system and the information presentation method according to the present invention described above can be realized by executing the information presentation program of the present invention described in a predetermined language on a computer. That is, the program of the present invention is installed in a mobile terminal device, a smart phone, a wearable terminal, a mobile PC or other information processing terminal, an IC chip or a memory device of a general-purpose computer such as a personal computer or a server computer, and executed on the CPU. Thus, it is possible to construct a system having the functions described above and implement the real estate evaluation method.

That is, a program that presents information on a virtual object placed in a virtual coordinate system of a virtual space,
A camera unit that captures images of real space;
A position information acquisition unit that acquires the position and direction in the coordinate system of the real space as current position information;
A real map acquisition unit for acquiring real map information including geographical information in the coordinate system of the real space;
A virtual map acquisition unit that acquires virtual map information in which the virtual object is arranged on geographical information in the coordinate system of the virtual space;
A virtual reality display data generation unit that generates virtual display data in which the virtual object is arranged on the virtual map information shared with the coordinate system in the real space;
Augmented reality display data for generating augmented display data by superimposing display data in which the virtual object is arranged on the real map information associated with a coordinate system in the real space on the video imaged by the camera unit A generator,
According to the current position information acquired by the position information acquisition unit, the virtual display data and the extended display data are selectively switched, or a part or all of the other display data is switched to one display data. A display control unit that executes display transition processing to be combined;
A display unit for displaying the virtual display data and the extended display data subjected to display transition processing by the display control unit;
It is made to function as.

  Such an information presentation program of the present invention can be distributed through a communication line, for example, and can be distributed as a package application that operates on a stand-alone computer by recording on a computer-readable recording medium. Can be transferred. Specifically, the recording medium can be recorded on various recording media such as a magnetic recording medium such as a flexible disk and a cassette tape, an optical disk such as a CD-ROM and a DVD-ROM, and a RAM card. According to the computer-readable recording medium on which the program is recorded, the above-described system and method can be easily implemented using a general-purpose computer or a dedicated computer, and the program can be stored, transported, and Easy installation.

  As described above, according to these inventions, in the content provided by information processing, the respective coordinate systems are integrated, and augmented reality (AR) and virtual reality (VR) coexist in one coordinate system and mixed. By doing so, it is possible to realize a new way of viewing and enjoyment by fusing augmented reality and virtual reality that have been separated and treated separately and conventionally. As a result, content (live-action / CG) that has not been experienced so far can be realized, and its use will be promoted in the field of industrial VR, which is expected to expand in the future, and in the amusement field such as games and attractions. Expected. As a result, according to the present invention, it is possible to provide a learning effect and the like by producing a virtual experience that is rich in diversity and high in reality.

It is a conceptual diagram which shows the whole structure of the system which concerns on embodiment. It is explanatory drawing which shows the relationship between the real map information M1 which concerns on embodiment, virtual map information M2, and an object, (a) shows virtual space, (b) shows real space, (c) shows augmented reality. Show. It is a block diagram which shows the outline | summary of the information presentation process which concerns on embodiment. It is a block diagram which shows the operation | movement in the information presentation process which concerns on embodiment. It is a block diagram which shows the internal structure of the content delivery server which concerns on embodiment. It is a block diagram which shows the internal structure of the content delivery server which concerns on embodiment. It is a sequence diagram which shows operation | movement of the information presentation system which concerns on embodiment. It is explanatory drawing regarding the boundary line of the display transition process which concerns on embodiment. It is explanatory drawing regarding the boundary surface of the display transition process which concerns on embodiment. It is a conceptual diagram which shows the condition which image | photographed the augmented reality display which concerns on embodiment. It is a conceptual diagram which shows the condition which image | photographed the virtual reality display which concerns on embodiment. It is explanatory drawing regarding trigger generation in the display transition process which concerns on embodiment. It is explanatory drawing which shows the outline | summary of the information presentation system which concerns on the example 1 of a change. It is explanatory drawing which shows the outline | summary of the information presentation system which concerns on the example 2 of a change.

(Overall configuration of information presentation system)
Embodiments of an information presentation system according to the present invention will be described below in detail with reference to the accompanying drawings. FIG. 1 is a conceptual diagram illustrating the overall configuration of the information presentation system according to the present embodiment, and FIG. 2 is an explanatory diagram illustrating the relationship between real map information M1, virtual map information M2, and objects according to the present embodiment. . 3 and 4 are block diagrams showing an outline and operation of the information presentation process according to the present embodiment.

  As shown in FIG. 2, the information presentation system according to the present embodiment, through an information terminal device such as a smartphone used by a user, and a virtual reality video (FIG. 2A) and a real space video (FIG. 2B). ) And augmented reality (FIG. 10 (c)) are switched or superimposed to present information about virtual objects arranged in a virtual coordinate system of the virtual space. Present as virtual reality (VR). In this embodiment, when displaying augmented reality or virtual reality, the coordinate systems are integrated so that the real coordinate system of augmented reality and the virtual coordinate system of virtual reality coexist in one coordinate system and mixed. can do. As a result, by combining AR and VR, it is possible to produce a virtual experience that is rich in diversity and highly realistic.

  Here, the basic processing of virtual reality and augmented reality will be described. The virtual reality (VR) shown in FIG. 2 (a) is a still image taken by a smartphone camera or the like as shown in FIG. 3DCG (Three-dimensional computer graphics) still image or CG animation object Ob1 is added to the real map information M1 of the real coordinate system (latitude, longitude, altitude) for the background and objects of pictures and videos. Based on the mixing processing (visible / invisible / transparent), it is displayed. On the other hand, the augmented reality (AR) shown in FIG. 2C is a virtual coordinate system in which a still image or CG animation object Ob1 as 3DCG is placed on the background of a still image or CG animation as 3DCG, as shown in FIG. Based on the virtual map information M2 of (x, y, z), a mixed process (visible / invisible / transparent) is displayed.

  In this mixing process, display control including composite image generation and display transition is performed based on scenario data and user operation, and layer control, dependency relations, grouping, and interlocking control are performed in display control. Further, when generating the augmented reality AR and virtual reality VR images, the camera position, the angle of view, the lighting, and the ambient light are referred to by referring to the time information acquired by the smartphone 1, the current position, orientation, illuminance, and the like. Are set based on each coordinate system.

  Specifically, as shown in FIG. 1, the information presentation system is roughly configured by a smartphone 1 that is a mobile terminal device used by a plurality of users 10 and a content distribution server 3 installed on the Internet 2. In the present embodiment, the smartphone 1 will be described as an example of an information processing terminal device.

  In the present embodiment, the content distribution server 3 is a content server that distributes information to each user terminal via the Internet 2, and can be realized by a single server device or a plurality of server device groups. Modules are virtually built on the CPU, and each functional module cooperates to execute processing. Further, the content distribution server 3 can perform data transmission / reception via the Internet 2 by a communication function, and can also present a web page through browser software by the web server function.

  The smartphone 1 is a portable information processing terminal device using wireless communication, and includes an arithmetic processing device such as a CPU and provides various functions by executing application software. As this information processing terminal device, for example, it can be realized by a general-purpose computer such as a personal computer or a dedicated device specialized in functions. In addition to a smartphone, a tablet PC, a mobile computer, a mobile phone, a wearable terminal device, a game And other mobile terminals.

  The smartphone 1 communicates wirelessly with a relay point such as the wireless base station 22 and can receive a communication service such as a call or data communication while moving. As communication methods for the telephone call and data communication, for example, 3G (3rd. Generation) method, LTE (Long Term Evolution) method, 4G method, FDMA method, TDMA method, CDMA method, W-CDMA, PHS (Personal) Handyphone System) method. The smartphone 1 is equipped with various functions such as a digital camera function, an application software execution function, and a position information acquisition function using GPS (Global Positioning System), and includes a mobile computer such as a tablet PC.

  The position information acquisition function provided in the smartphone 1 is a function for acquiring and recording position information indicating the position of the own device. As the position information acquisition function, for example, as shown in FIG. As described above, there are a method of detecting the position of the own device based on a signal from the satellite 21, and a method of detecting the position based on the radio wave intensity from a wireless base station 22 of a mobile phone or an access point of WiFi communication.

  The smartphone 1 includes a display unit such as a liquid crystal display for displaying information, and an operation device such as an operation button for a user to perform an input operation. The operation device is superimposed on a liquid crystal display. A touch panel is included as an input unit that is arranged and acquires an operation signal by a touch operation or the like that specifies a coordinate position on a liquid crystal display. Specifically, this touch panel is an input device that inputs an operation signal by pressure by a touch operation using a user's fingertip, pen, etc., and a liquid crystal display that displays a graphic and a coordinate position of the graphic displayed on the liquid crystal display And a touch sensor that receives an operation signal corresponding to is superimposed.

(Internal structure of each device)
Next, the internal structure of each device constituting the information presentation system described above will be described. FIG. 5 is a block diagram illustrating an internal configuration of the content distribution server 3 according to the present embodiment, and FIG. 6 is a block diagram illustrating an internal configuration of the smartphone 1 according to the present embodiment. The “module” used in the description refers to a functional unit that is configured by hardware such as an apparatus or a device, software having the function, or a combination thereof, and achieves a predetermined operation. .

(1) Content Distribution Server First, the internal configuration of the content distribution server 3 will be described. The content distribution server 3 is a server device arranged on the Internet 2, and can send and receive data to and from each smartphone 1 through the Internet 2.

  The content distribution server 3 includes a communication interface 31 that performs data communication through the Internet 2, an authentication unit 33 that authenticates the authority of the user and the user terminal, and position information management that collects and manages position information of each user terminal. Unit 32, information management unit 36 that executes scenario-based content control, virtual map information generation unit 37 that generates virtual map information, data 34 that distributes various information to each user, and data that stores various data A storage unit is provided.

  The data storage unit includes real map data 35a serving as a real map storage unit that stores real map information including geographical information in the real world, user data 35b that stores information about users, virtual map information, and trigger conditions. And virtual content data 35c for storing scenario data. Each of these data may be a single table data, and can be divided into a plurality of databases and a relational database in which the respective data are linked by setting relations with each other.

  The real map data 35a is a storage device that stores real map information including geographical information in the real world, such as natural geographical elements such as mountains, valleys, and rivers, artifacts such as buildings, roads, and railways, and place names. And addresses, traffic regulations, etc. are stored. The real map data 35a may be a map database operated by another map service provider, in addition to the data owned and operated by the service provider operating the content distribution server 3 itself.

  The information stored in the user data 35b includes authentication information in which an identifier (user ID, terminal ID) that identifies a user or a mobile terminal device used by the user, a password, and the like is associated with the user. The personal information of the user associated with the ID, the model of the terminal device, and the like are also included. In addition, the user data 35b includes authentication history (access history) for each user or user terminal, information on scenario data for each user based on the relation with the virtual content data 35c, payment information on a content providing service, and the like. Accumulated.

  Information stored in the virtual content data 35c includes virtual world map information, characteristics of characters and objects, scenario data, graphic information, and the like as virtual content data. The virtual content data is associated with geographical elements, buildings, roads, railways, and the like included in the real map, and includes coordinate information of boundary lines / boundary surfaces that are trigger conditions for display transition processing.

  The authentication unit 33 is a module that establishes a communication session with each smartphone 1 through the communication interface 31 and performs an authentication process for each established communication session. As this authentication process, authentication information is acquired from the smartphone 1 of the user who is an accesser, the user is identified with reference to the user data 35b, and their authority is authenticated. The authentication result (user ID, authentication time, session ID, etc.) by the authentication unit 33 is sent to the information management unit 36 and stored as authentication history in the user data 35b. In this embodiment, an authentication unit for specifying an accessor is provided. However, this authentication unit may be omitted, and a service may be provided to an unspecified accessor.

  The location information management unit 32 is a module that acquires location information acquired on the user terminal device side and transmitted to the content distribution server 3. The location information management unit 32 is specified by the authentication process by the authentication unit 33. The identifiers (user ID, terminal ID, etc.) of the user and user terminal device and their position information are linked and stored as usage history in the user data 35b.

  The information management unit 36 is a module that manages display transition processing that occurs when each user moves in the real world or the virtual world. In this embodiment, the information management unit 36 cooperates with the display control unit 143 on the smartphone 1 side, and performs part of the display data generation processing on the content distribution server 3 side to perform graphic processing or A part of the event processing is executed by the display control unit 143 on the smartphone 1 side. For example, the content distribution server 3 side predicts a display transition process that can occur based on the position of the object, the boundary line or the coordinate position of the boundary surface, and the generation condition is generated on the content distribution server 3 side. The conditions are transmitted to the smartphone 1 side, and the actual display transition process and the graphic process therefor are executed on the smartphone 1 side based on the generation conditions received from the content distribution server 3.

The virtual map information generation unit 37 sets the coordinate information of the object Ob1 and the like based on the virtual coordinate system corresponding to the geographic information on the real map information M1, and the boundary that causes the display transition to occur, according to the user's current position and scenario data settings. It is a module that generates virtual map information M2 including a line L1 or a boundary surface F1. More specifically, the virtual map information generation unit 37
In addition to acquiring geographical elements, buildings, roads, railways, and the like included in the real map information M1 accumulated in the virtual map, the information stored in the virtual content data 35c is referred to, and the virtual world corresponding to the real map information M1 is obtained. Based on map information, object characteristics, display transition processing information, graphic information, etc., virtual world map information is generated. Here, as shown in FIGS. 8 and 9, the virtually restored old castle object Ob1 is arranged at the coordinate position (coordinate range) corresponding to the area P1 which is the ruins on the real map information M1. .

  Based on the current location of the user, the information distribution unit 34 transmits the virtual world graphic generated by the virtual map information generation unit 37 and the real map information stored in the real map data 35a to each user via the communication interface 31. It is a module to be distributed to.

(2) Smartphone 1
Next, the internal configuration of the smartphone 1 will be described. As shown in FIG. 6, the smartphone 1 includes a communication interface 11, an input interface 12, an output interface 13, an application execution unit 14, and a memory 15.

  The communication interface 11 is a communication interface for performing data communication, and has a function of performing contact (wired) communication by wireless non-contact communication or a cable, adapter means, or the like. The input interface 12 is a device to which a user operation such as a mouse, a keyboard, operation buttons, and a touch panel 12a, a still image or a moving image in a real space captured by the camera unit 121 is input. The output interface 13 is a device that outputs video and sound, such as a display and a speaker. In particular, the output interface 13 includes a display unit 13a such as a liquid crystal display, and the display unit 13a is superimposed on a touch panel 12a that is an input interface.

  The memory 15 is a storage device that stores an OS (Operating System), firmware, programs for various applications, other data, and the like. The memory 15 includes various IDs for identifying the user or the smartphone 1. The application data downloaded from the content distribution server 3 is accumulated, and various data processed by the application execution unit 14 are accumulated. Furthermore, the virtual map information M2 and the real map information M1 acquired from the content distribution server 3 are stored in the memory 15 according to the present embodiment.

  The application execution unit 14 is a module that executes an application such as a general OS, various applications, and browser software, and is usually realized by a CPU or the like. The application execution unit 14 executes the information presentation program according to the present invention, so that the information acquisition unit 141, the display data generation unit 142, the display control unit 143, the scenario management unit 144, and the trigger determination unit 145 are executed. The transition setting unit 146 and the position information acquisition unit 147 are virtually constructed.

  The information acquisition unit 141 acquires real map information including geographical information in the real space coordinate system, and virtual map information in which virtual objects are arranged on the geographical information in the virtual space coordinate system. This module plays the role of a map acquisition unit. The information acquisition unit 141 acquires information when reading data stored in the memory 15, downloading from the content distribution server 3 through the communication interface 11, or generated or processed in the smartphone 1. Any case of reading data is included. Display processing of the display data generated by the display data generation unit 142 is controlled by the display control unit 143.

  The display data generation unit 142 is a module that generates display data to be displayed on the display unit 13a. The display data is data generated by combining image data, character data, moving image data, audio and other data in addition to 3D graphic data. In particular, the display data generation unit 142 uses virtual display data in which a virtual object Ob1 that is a 3D graphic is arranged on the virtual map information M2 that is shared with the coordinate system in the real space and associated with the position information in the real space. It is a module that plays the role of a virtual reality display data generation unit to be generated. In addition, the display data generation unit 142 converts the display data in which the virtual object Ob1 that is a 3D graphic is arranged on the real map information associated with the position information of the coordinate system in the real space into an image captured by the camera unit 121. It plays the role of an augmented reality display data generation unit that generates augmented display data by superimposing.

  The display control unit 143 selectively switches between the virtual display data and the extended display data 0 according to the current position information acquired by the position information acquisition unit 147, or switches one display data to one of the other display data. This is a module for executing a display transition process for combining parts or all. The virtual display data or the extended display data subjected to the display transition process by the display control unit 143 is displayed on the display unit 13a through the output interface 13.

  The display control unit 143 also has a function of synchronizing processing on the smartphone 1 side with processing on the content distribution server 3 side. Specifically, the content distribution server 3 predicts a display transition process that can occur based on the position of the smartphone 1 or the position of the object Ob1, and virtually sets a trigger condition that is a generation condition of the display transition process. It adds to map information and transmits the virtual map information to the smartphone 1 side. This trigger condition is acquired by the transition setting unit 146, and the actual display transition processing and the graphic processing therefor are performed based on the trigger condition received from the content distribution server 3, the trigger determination unit 145 on the smartphone 1 side, display data generation Processing is performed by the unit 142 and the display control unit 143. The result of the display transition process executed by the trigger determination unit 145 and the display control unit 143 is notified to the information management unit 36 on the content distribution server 3 side and is reflected in the subsequent process in the virtual map information generation unit 37.

  The scenario management unit 144 is a module that acquires and manages scenario data describing the time at which the display transition process is executed, the place, the types of objects to be presented, and the order of their appearance. The display control unit 143 is a trigger determination This is a module that executes display transition processing according to scenario data or user operation based on the determination result by the unit 145.

  The position information acquisition unit 147 is a module that acquires the position and direction in the coordinate system of the real space as current position information. Specifically, the position information acquisition unit 147 uses the position information acquisition function, gyro sensor, azimuth sensor, acceleration sensor, and the like provided in the smartphone 1 to calculate the current position of the own device based on the GPS and the surrounding radio wave environment. The altitude, the heading of the aircraft, the relative movement distance, and the relative rotation angle are acquired. Then, the coordinate position in the real space and the coordinate position in the virtual space are mutually converted, and when the display control unit 143 displays augmented reality, the coordinate position in the real space is sent to the trigger determination unit 145 and the display control unit 143. When the display control unit 143 displays the virtual reality, the coordinate position in the real space is converted into the coordinate position on the virtual space map and is sent to the trigger determination unit 145 and the display control unit 143.

  The trigger determination unit 145 is a module that determines that the position information included in the current position information has passed a boundary line or a boundary plane set on the real space map or the virtual space map. Specifically, as shown in FIG. 12, the display control unit 143 displays the augmented reality with reference to the current position in the real space acquired by the position information acquisition unit 147 or the coordinate position on the virtual space map. Sometimes, the coordinate position of the real space is monitored, and when the movement of the user passes the boundary line L1 or the boundary surface F1, a trigger signal is output and sent to the display control unit 143, while the display control unit 143 displays the virtual reality. The virtual coordinate position obtained by converting the coordinate position of the real space into the coordinate system on the virtual space map is monitored, and triggered when the user's movement passes the boundary line L1 or the boundary plane F1 on the virtual map. The signal is output and sent to the display control unit 143.

  At this time, when the scales of the coordinate system of the real map information M1 and the coordinate system of the map M2 are equivalent, the distance moved in the real space (augmented reality space) is unchanged as shown in FIG. It becomes a moving distance in the virtual space, a trigger is generated by passing through the boundary line L1 or the boundary surface F1 in the real space (virtual reality space), the virtual space is displayed by the display transition, and walking in the real space The movement distance becomes the movement distance in the virtual space as it is, and a trigger is generated when the movement in the virtual space passes through the boundary line L1 or the boundary surface F1, and the display transition is executed to display the real space. Be started.

  Further, when the scale of the real map information M1 is different from the scale of the map M2, as shown in FIG. 12B or 12C, a distance different from the distance moved in the real space (augmented reality space). Is the moving distance in the virtual space. In FIG. 5B, for example, the user's body is enlarged and moved within the virtual space a distance that is larger than the distance actually walked, and the boundary line L1 or the boundary is moved by the movement in the enlarged virtual space. A trigger is generated by passing through the plane F1, display transition is executed, and display of the real space is started. In FIG. 8C, for example, the user's body is reduced and moved within the virtual space by a distance that is reduced compared to the actual walking distance, and the boundary line L1 or the boundary is moved by the movement in the reduced virtual space. A trigger is generated by passing through the plane F1, display transition is executed, and display of the real space is started.

  Also, when the user moves on a virtual vehicle in the virtual space, for example, as shown in FIG. 12D, the user does not move while driving the vehicle while the virtual space is displayed. -Moves in the virtual space by maneuvering, triggers by passing through the boundary line L1 or the boundary surface F1 due to the movement in the virtual space, display transition is executed, and display of the real space starts Is done.

  The transition setting unit 146 is a module that sets the boundary line L1 or the boundary surface F1 for executing the display transition process on the real map information M1 or the virtual map information M2 as shown in FIGS. Here, the boundary line is a straight line or a curve expressed on a two-dimensional map coordinate system as shown in FIGS. 8A and 8B, and as shown in FIGS. 9A and 9B. The boundary surface is a plane or a curved surface expressed on a three-dimensional map coordinate system. In FIG. 8, the planar shape of the area P1 and the object Ob1 is defined by being surrounded by the boundary line L1, and in FIG. 9, the polygon that is surrounded by the boundary surface F1 and forms the outer surface of the three-dimensional area P1 and the object Ob1. Is defined.

  Specifically, the transition setting unit 146 extracts the boundary line L1 or the boundary surface F1 on the real map information M1 or the virtual map information M2 acquired by the information acquisition unit 141, and the boundary line L1 is infinite in the three-dimensional space. The plane is set as a plane having a distant height, while the boundary plane F1 is set as a boundary line on a two-dimensional map according to the altitude of the current position. For example, when the boundary surface F1 is set on a predetermined floor in a building having a plurality of floors, it appears on a two-dimensional planar map according to the height (elevation) of the current position. A boundary line that is a cross section of the boundary surface F1 is set on the planar map of each floor.

  The display control unit 143 is a module that executes display transition processing based on the determination result by the trigger determination unit 145. In addition, the display control unit 143 cooperates with the information management unit 36 on the content distribution server 3 side, and performs part of the display data generation processing on the content distribution server 3 side to perform graphic processing and the like. A part is executed by the display data generation unit 142 on the smartphone 1 side. For example, display transition conditions, display data, and the like are generated on the content distribution server 3 side, the conditions and display data are transmitted to the smartphone 1 side, and the actual display transition processing and graphic processing therefor are performed on the smartphone 1 side. May be executed.

  In the system having such a configuration, the virtual reality (VR) shown in FIG. 2A is a background of a still image or a moving image that is a photograph taken with a camera of a smartphone, as shown in FIG. The object acquisition unit 141 acquires a 3DCG still image or CG animation object Ob1 as an object, and the display data generation unit 142 sets the 3DCG object Ob1 as a real coordinate system (latitude, longitude, altitude). ) Is generated as an image for each layer. The image for each layer is mixed in the display control unit 143 based on the layer control by the scenario management unit 144 and displayed on the touch panel 12a. On the other hand, in the augmented reality (AR) shown in FIG. 2C, as shown in FIG. 4, a 3DCG still image or CG animation object Ob1 is used as an information acquisition unit in the background of a 3DCG still image or CG animation. 141, and the display data generation unit 142 uses the 3DCG still image or CG animation background as the background image of the 3DCG still image or CG animation object Ob1 in the virtual coordinate system (x, y, z) virtual map information M2. Is generated as an image for each layer. The image for each layer is mixed in the display control unit 143 based on the layer control by the scenario management unit 144 and displayed on the touch panel 12a.

  In this mixing process, display control including composite image generation, display transition, and the like is performed based on scenario data managed by the scenario management unit 144 and user operations acquired through a user interface such as the touch panel 12a. In the control, layer control, dependency relationship, grouping, and interlock control are performed. In addition, when generating the augmented reality AR and virtual reality VR images, the time information acquired by the various sensors 16 and the clock unit included in the smartphone 1, the current position, direction, illuminance, and the like are referred to, and the camera The position, angle of view, lighting, and ambient light are set based on each coordinate system.

(Operation of information presentation system)
By operating the information presentation system described above, the information presentation method of the present invention can be implemented. FIG. 7 is a sequence diagram showing operations related to display transition processing of the information presentation system.

  As shown in FIG. 10, for example, an object of an old castle formed on the coordinates of the virtual map information M2 corresponding to the area P1 that is the ruins approaching the predetermined area P1 such as the ruins on the real map information M1 When Ob1 enters the field of view of the camera, the augmented reality display is started by the trigger determination unit 145, and the appearance of the object Ob1, which is a virtual building with an actual landscape as a background, can be viewed as augmented reality.

  Next, when the vehicle further approaches the area P1 and passes through the boundary line L1 or the boundary surface F1 and enters the area P1 as shown in FIG. 11, the display transition process is executed assuming that the trigger condition is satisfied, and the virtual The inside of the object Ob1, which is a typical building, can be viewed as a virtual reality.

  Such display transition processing will be specifically described. As shown in FIG. 7, first, when presenting information, each smartphone 1 periodically acquires position information and transmits it to the content distribution server 3 (S201 and S101). On the content distribution server 3 side, the display transition process to be generated is predicted according to the transmitted current position of each user (S102), and the generation condition is transmitted to the smartphone 1 side (S103).

  More specifically, based on the current position of each user acquired on the content distribution server 3 side, the information management unit 36 of the content distribution server 3 can generate based on the current position of the user, the position of the object, and the like. The display transition process is predicted, the generation condition is generated on the content distribution server 3 side, and the condition is transmitted to the smartphone 1 side.

  On the smartphone 1 side, this trigger condition is acquired by the transition setting unit 146. When shooting by the camera unit 121 is started (S203), the trigger determination unit 145 monitors the timing at which the trigger condition is satisfied (S204). Specifically, a trigger condition that monitors the trigger condition, which is a condition for generating the display transition process, and the current position of the own device acquired by the position information acquisition unit 147 on the smartphone 1 side and enters the area P1 is set. When it is satisfied (“Y” in S205), a display transition process is executed (S206). On the other hand, if the trigger condition is not satisfied (“N” in S205), the imaging process (S203) and the monitoring of the trigger condition (S204) are continued as they are.

  More specifically, in the monitoring of the trigger condition, the trigger determination unit 145 indicates that the position information included in the current position information has passed the boundary line or boundary surface set on the real space map or the virtual space map. judge. For example, as shown in FIG. 12, when the display control unit 143 displays augmented reality with reference to the current position on the real space acquired by the position information acquisition unit 147 or the coordinate position on the virtual space map, the real space is displayed. The coordinate position is monitored, and when the movement of the user passes through the boundary line L1 or the boundary surface F1, a trigger signal is output and sent to the display control unit 143. On the other hand, when the display control unit 143 displays the virtual reality, the virtual coordinate position obtained by converting the coordinate position of the real space into the coordinate system on the virtual space map is monitored, and the movement of the user is the boundary line on the virtual map. When the signal passes through L1 or the boundary surface F1, a trigger signal is output and sent to the display control unit 143.

  At this time, when the scales of the coordinate system of the real map information M1 and the coordinate system of the map M2 are equivalent, the distance moved in the real space (augmented reality space) is unchanged as shown in FIG. It becomes a moving distance in the virtual space, a trigger is generated by passing through the boundary line L1 or the boundary surface F1 in the real space (virtual reality space), the virtual space is displayed by the display transition, and walking in the real space The movement distance becomes the movement distance in the virtual space as it is, and a trigger is generated when the movement in the virtual space passes through the boundary line L1 or the boundary surface F1, and the display transition is executed to display the real space. Be started.

  Further, when the scale of the real map information M1 is different from the scale of the map M2, as shown in FIG. 12B or 12C, a distance different from the distance moved in the real space (augmented reality space). Is the moving distance in the virtual space. In FIG. 5B, for example, the user's body is enlarged and moved within the virtual space a distance that is larger than the distance actually walked, and the boundary line L1 or the boundary is moved by the movement in the enlarged virtual space. A trigger is generated by passing through the plane F1, display transition is executed, and display of the real space is started. In FIG. 8C, for example, the user's body is reduced and moved within the virtual space by a distance that is reduced compared to the actual walking distance, and the boundary line L1 or the boundary is moved by the movement in the reduced virtual space. A trigger is generated by passing through the plane F1, display transition is executed, and display of the real space is started.

  Also, when the user moves on a virtual vehicle in the virtual space, for example, as shown in FIG. 12D, the user does not move while driving the vehicle while the virtual space is displayed. -Moves in the virtual space by maneuvering, triggers by passing through the boundary line L1 or the boundary surface F1 due to the movement in the virtual space, display transition is executed, and display of the real space starts Is done.

Thereafter, the result of the display transition process executed on the smartphone 1 side is notified to the information management unit 36 on the content distribution server 3 side through the display control unit 143 (S207), and the synchronization process is performed on the content distribution server 3 side ( S104), the result of the executed display transition process is reflected in the scenario generation process.
The above processing is repeated until the application is terminated.

(Action / Effect)
According to the present embodiment described above, when displaying augmented reality (AR) and virtual reality (VR), by integrating the respective coordinate systems, augmented reality (AR) and virtual reality (VR) are combined into one. Can coexist in the coordinate system and mix. As a result, by combining AR, it is possible to produce a virtual experience that is rich in diversity and highly realistic.

  Further, according to the present embodiment, augmented reality and virtual reality can be obtained without requiring a special operation by the user by passing through a boundary line or boundary surface set on the real map information or virtual map information. It is possible to switch automatically and to make a natural transition from augmented reality to virtual reality or from virtual reality to augmented reality.

(Example of change)
The above description of the embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made according to the design and the like as long as they do not depart from the technical idea of the present invention. 13 and 14 are explanatory diagrams showing an overview of the information presentation system according to the first and second modification examples.

(1) Modification 1
For example, in the above-described embodiment, the smartphone 1 is used as the information processing apparatus including the camera unit. However, as the apparatus including the camera unit, a viewing mirror 40 installed in a sightseeing spot as illustrated in FIG. 13 is used. It may be used. The observing mirror 40 has a base 40b fixed in a real space coordinate system, and the observing mirror main body 40a is installed so as to be turnable to the left and right and up and down with respect to the base 40b. Yes. The camera unit 121 is disposed at the front end of the viewing mirror body 40a, and the display unit 13a is disposed at the rear end side of the viewing mirror body 40a.

  In this modified example 1, as shown in FIG. 13, when viewing the area P1 that is the ruins of the old castle in the real space through the viewing mirror 40, the coordinate position (coordinate range) corresponding to the area P1 that is the ruins is It is possible to view an augmented reality image in which the virtually restored old castle object Ob1 is combined with a real space background. The object Ob1, which is an augmented reality video, is three-dimensionally adapted to the terrain and coordinates of the background of the real space that can be photographed by the observing mirror 40, and rotates the observing mirror main body 40a to the left and right or up and down. By doing so, you can watch the ancient castle that was virtually restored and the stereoscopic video that combines the surrounding background. At this time, the passage of the boundary line or the boundary surface may be determined by the movement of the focal length of the viewing mirror, and the display transition may be triggered. According to the first modification, it is possible to add a virtual or expansive effect such as the restored old castle and the scenery at that time to the scenery of the real space seen with the observing mirror.

(2) Modification example 2
Further, as a 3D object synthesized with virtual reality or augmented reality, an object Ob2 as a so-called face-shaped face signboard having an opening as shown in FIG. 14 may be used. For example, when the subject Sub1 is photographed by the smartphone 1 with the area P1 that is the ruins of an old castle ruin in the real space, the object Ob2 as a face saddle sign is augmented realistically at the coordinate position between the subject Sub1 and the smartphone 1. The old castle object Ob1 is arranged in augmented reality behind the subject Sub1.

  The object Ob2 as the face saddle sign is distributed from the content distribution server 3 with the content distribution server 3 selecting an object corresponding to the current position. According to this modified example 2, it is possible to synthesize a still picture or a moving picture with a unique face design along with the restored old castle and the scenery at that time. A unique virtual or expansion effect can be added.

F1 ... Boundary surface L1 ... Boundary line M2 ... Virtual map information M1 ... Real map information P1 ... Remains 1 ... Smartphone 2 ... Internet 3 ... Content distribution server 10 ... User 11 ... Communication interface 12 ... Input interface 12a ... Touch panel 13 ... Output interface 13a ... display unit 14 ... application execution unit 15 ... memory 16 ... seed sensor 21 ... satellite 22 ... radio base station 31 ... communication interface 32 ... location information management unit 33 ... authentication unit 34 ... information distribution unit 35a ... real map data 35b ... User data 35c ... Virtual content data 36 ... Information management unit 37 ... Virtual map information generation unit 40 ... Viewing mirror 40a ... Observation mirror body 40b ... Base unit 121 ... Camera unit 141 ... Information acquisition unit 142 ... Display data generation unit 143 ... Display control Unit 144 ... Scenario management unit 145 ... Trigger determination unit 146 ... Transition setting unit 147 ... Position information acquisition unit

Claims (7)

A system for presenting information about a virtual object arranged in a virtual coordinate system of a virtual space,
A camera unit that captures images of real space;
A position information acquisition unit that acquires the position and direction in the coordinate system of the real space as current position information;
A real map acquisition unit for acquiring real map information including geographical information in the coordinate system of the real space;
A virtual map acquisition unit that acquires virtual map information in which the virtual object is arranged on geographical information in the coordinate system of the virtual space;
A virtual reality display data generation unit that generates virtual display data in which the virtual object is arranged on the virtual map information shared with the coordinate system in the real space;
Augmented reality display data for generating augmented display data by superimposing display data in which the virtual object is arranged on the real map information associated with position information in the real space on the video imaged by the camera unit A generator,
According to the current position information acquired by the position information acquisition unit, the virtual display data and the extended display data are selectively switched, or a part or all of the other display data is switched to one display data. A display control unit that executes display transition processing to be combined;
An information presentation system comprising: a display unit that displays virtual display data and extended display data subjected to display transition processing by the display control unit. A transition setting unit for setting a boundary line or a boundary surface for executing the display transition process on the real map information or the virtual map information;
A trigger determination unit that determines that the position information included in the current position information has passed the boundary line or boundary surface;
The information display system according to claim 1, wherein the display control unit executes the display transition process based on a determination result by the trigger determination unit. A scenario management unit for acquiring and managing scenario data describing the time, place, type of objects to be presented, and the order of their appearance, for executing the display transition process;
The information display system according to claim 1, wherein the display control unit executes the display transition process according to the scenario data or a user operation based on a determination result by the trigger determination unit. The base part fixed to the coordinate system of the real space further comprises a main body part that can be turned left and right and can be lifted up and down,
The camera part is disposed at the tip of the main body part,
The information display system according to claim 1, wherein the display unit is arranged at a rear end of the main body unit.   The information presentation system according to claim 1, wherein the object is a plate-shaped signboard having an opening. A method for presenting information about a virtual object placed in a virtual coordinate system of a virtual space,
A real space information acquisition step in which the camera unit captures an image of real space, and the position information acquisition unit acquires the position and direction in the coordinate system of the real space as current position information;
Real map information including geographic information in the coordinate system of the real space and virtual map information in which the virtual object is arranged on the geographic information in the coordinate system of the virtual space are acquired, and a virtual display data generation unit is configured to acquire the real space. And generating virtual display data in which the virtual object is arranged on the virtual map information that is shared with the coordinate system in FIG. 5 and the extended display data generating unit is associated with the position information in the real space. A display data generation step of generating extended display data by superimposing display data in which the virtual object is arranged on the information on the video captured by the camera unit;
According to the current position information acquired by the position information acquisition unit, the virtual display data and the extended display data are selectively switched, or a part or all of the other display data is switched to one display data. A display control step in which the display control unit executes display transition processing to be combined; and
A display step in which the display control unit displays the virtual display data and the extended display data subjected to the display transition process in the display control step;
The information presentation method characterized by including. A program that presents information about a virtual object placed in a virtual coordinate system of a virtual space,
A camera unit that captures images of real space;
A position information acquisition unit that acquires the position and direction in the coordinate system of the real space as current position information;
A real map acquisition unit for acquiring real map information including geographical information in the coordinate system of the real space;
A virtual map acquisition unit that acquires virtual map information in which the virtual object is arranged on geographical information in the coordinate system of the virtual space;
A virtual reality display data generation unit that generates virtual display data in which the virtual object is arranged on the virtual map information shared with the coordinate system in the real space;
Augmented reality display data for generating augmented display data by superimposing display data in which the virtual object is arranged on the real map information associated with position information in the real space on the video imaged by the camera unit A generator,
According to the current position information acquired by the position information acquisition unit, the virtual display data and the extended display data are selectively switched, or a part or all of the other display data is switched to one display data. A display control unit that executes display transition processing to be combined;
A display unit for displaying the virtual display data and the extended display data subjected to display transition processing by the display control unit;
An information presentation program characterized by functioning as