JP2022126546A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To provide a technology for identifying a relationship between an AR (Augmented Reality) object and an environment where the AR object is disposed.SOLUTION: A specification information acquisition unit 110 acquires specification information for specifying data including a three-dimensional model of a building in a real world. A matching unit 111 superimposes the three-dimensional model included in the data specified by the specification information on an image being captured by an imaging unit 12. A display control unit 112 causes a display unit 13 to display the captured image with the three-dimensional model superimposed thereon. An object management unit 113 receives a selection of an AR object to be disposed in a space which is specified by a three-dimensional coordinate system defining the three-dimensional model, and settings for a position and direction to dispose the AR object in the three-dimensional coordinate system. The display control unit 112 causes the display unit 13 to display the selected AR object in the set position and direction.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device, an information processing method, and a program.

In recent years, technologies called Augmented Reality (AR) and Mixed Reality (MR) have been rapidly developing, and are being put to practical use in daily life. These technologies use smartphones and smart glasses to superimpose and display virtual objects (hereinafter referred to as “AR objects” in this specification) on images of real objects, This is a technique for presenting information to the user (see Patent Document 1, for example).

Japanese Patent Application Laid-Open No. 2020-57430

Patent Literature 1 discloses a technique for presenting an AR object having the same size as a relatively large structure such as a station, a building, or an urban area at a location where the structure exists. With such technology, of course, the user can observe the AR object only after going to the city where the AR object is placed. Therefore, for example, even those who decide the location of the AR object cannot confirm how the AR object looks in relation to the structure unless they actually go to the site.

The present invention has been made in view of these points, and it is an object of the present invention to provide a technique for grasping the relationship between an AR object and the environment in which the AR object is arranged.

A first aspect of the present invention is an information processing device. This device includes a specific information acquisition unit that acquires specific information for specifying data including a three-dimensional model of a structure in the real world, and a captured image being captured by an imaging unit that acquires data specified by the specific information. a matching unit that superimposes the included three-dimensional model; a display control unit that causes a display unit to display the captured image superimposed with the three-dimensional model; and a three-dimensional coordinate system that defines the three-dimensional model. an object management unit that receives selection of an AR object to be installed in space and setting of the installation position and installation orientation of the AR object in the three-dimensional coordinate system; The display control unit causes the display unit to display the selected AR object at the installation position in the installation orientation.

The information processing device further includes a communication unit that transmits object information including information specifying the selected AR object and information indicating settings of the installation position and installation orientation of the AR object to the content server. may

The specific information acquisition unit may acquire information indicating a position where the structure exists in the real world from the user as the specific information.

The matching unit matches a plane specified by two axes in a three-dimensional coordinate system that defines the three-dimensional model with a planar object included in the captured image, thereby matching the captured image and the three-dimensional model. A model may be superimposed.

A second aspect of the present invention is an information processing method. In this method, the processor obtains specific information for specifying data including a three-dimensional model of a structure in the real world; superimposing the included three-dimensional model; displaying the captured image superimposed with the three-dimensional model on a display unit; and installing the three-dimensional model in a space specified by a three-dimensional coordinate system a step of receiving a selection of an AR object for displaying and setting of an installation position and an installation orientation of the AR object in the three-dimensional coordinate system; and causing the installation position to display in the installation orientation.

A third aspect of the present invention is a program. This program provides a computer with a function of acquiring specific information for specifying data including a three-dimensional model of a structure in the real world, and acquiring data specified by the specific information in a captured image being captured by an imaging unit. A function of superimposing the included three-dimensional model, a function of displaying the captured image superimposed with the three-dimensional model on a display unit, and installation in a space specified by a three-dimensional coordinate system that defines the three-dimensional model. a function of receiving a selection of an AR object to be used, a setting of the installation position and installation orientation of the AR object in the three-dimensional coordinate system, and displaying the selected AR object on the display unit. and a function of displaying at the installation position in the installation orientation.

In order to provide this program or update part of the program, a computer-readable recording medium recording this program may be provided, or this program may be transmitted via a communication line.

Any combination of the above-described components, and expressions of the present invention converted into methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to provide a technique for understanding the relationship between an AR object and the environment in which the AR object is arranged.

FIG. 2 is a diagram for explaining an outline of processing executed by an information processing apparatus according to an embodiment; FIG. 1 is a diagram schematically showing the configuration of a display system according to an embodiment; FIG. FIG. 4 is a diagram schematically showing an example of a specific information specifying screen provided by the information processing device according to the embodiment; FIG. 4 is a diagram for explaining matching between a real plane object and a city object; FIG. 10 is a diagram schematically showing an example of a selection screen for selection objects provided by the information processing apparatus according to the embodiment; 4 is a flowchart for explaining the flow of information processing executed by the information processing apparatus according to the embodiment;

<Overview of Embodiment>
FIGS. 1(a) to 1(d) are diagrams for explaining an outline of processing executed by an information processing apparatus according to an embodiment. An outline of the embodiment will be described below with reference to FIG.

An information processing apparatus according to an embodiment, for example, displays an image of an existing city on an existing object having a planar structure like a desk in a user's home (hereinafter referred to as a "real planar object"). A three-dimensional model AR object (hereinafter referred to as a “city object”) is superimposed and displayed on the display unit. FIG. 1(a) is a diagram schematically showing an existing desk. In the example shown in FIG. 1(a), a cup R2 and a notebook R3 are placed on a real plane object R1, which is a real desk.

FIG. 1(b) shows the plane P detected in the real plane object R1 by the information processing device. In FIG. 1B, the area indicated by the dashed rectangle is the plane P detected on the real plane object R1. Also, FIG. 1(c) shows an example of a city object V superimposed on the plane P of the real plane object R1. As shown in FIG. 1(c), the user observing the display unit can observe an image as if a city diorama exists on the real plane object R1.

When the user selects an AR object (hereinafter referred to as "selected object") to be superimposed on the city object V via the operation unit of the information processing device and designates an arrangement location, etc., the information processing device displays the city object An image in which the selected object is superimposed on V is further superimposed on the real plane object R1 and displayed on the display unit. FIG. 1(d) is an enlarged view of the city object V, and specifically shows a state in which the selected object A is superimposed on the city object V. As shown in FIG.

FIG. 1(d) shows an example in which the user selects a penguin character as the selection object A. FIG. In order to avoid complication, not all selected objects A are labeled, but in FIG. FIG. 1(d) shows how four selected objects are superimposed on the city object V. FIG.

As described above, the information processing apparatus according to the embodiment arranges a city object that simulates an existing city on the real plane object R1, and provides the user with an image in which the selected object is further superimposed on the city object. As a result, the information processing apparatus according to the embodiment allows the user to grasp the relationship between the selection object, which is an AR object, and the environment of the city in which the selection object is arranged, without actually visiting the city where the user actually exists. can.

<Functional Configuration of Information Processing Apparatus 1 According to Embodiment>
FIG. 2 is a diagram schematically showing the configuration of the display system S according to the embodiment. A display system S includes an information processing device 1 , a content server 2 and a model server 3 . The information processing device 1 is connected to the content server 2 and the model server 3 via a network N such as the Internet in a communicable manner, and is a portable terminal such as a smart phone. 2 also shows the functional configuration of the information processing apparatus 1. As shown in FIG. The information processing device 1 includes a storage unit 10 , a control unit 11 , an imaging unit 12 , a display unit 13 and a communication unit 14 .

In FIG. 2, arrows indicate main data flows, and there may be data flows not shown in FIG. In FIG. 2, each functional block does not show the configuration in units of hardware (apparatus), but the configuration in units of functions. Therefore, the functional blocks shown in FIG. 2 may be implemented within a single device, or may be implemented separately within a plurality of devices. Data exchange between functional blocks may be performed via any means such as a data bus, network, or portable storage medium.

The storage unit 10 includes a ROM (Read Only Memory) that stores a BIOS (Basic Input Output System) of a computer that implements the information processing apparatus 1, a RAM (Random Access Memory) that serves as a work area of the information processing apparatus 1, an OS ( (Operating System), application programs, and a large-capacity storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information referred to when the application program is executed.

The control unit 11 is a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the information processing device 1, and executes a program stored in the content server 2 to obtain the specific information acquisition unit 110, the matching unit 111 , a display control unit 112 , and an object management unit 113 .

The imaging unit 12 is a camera for imaging the environment outside the information processing device 1, and uses a known solid-state imaging device such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. has been realized. The display unit 13 is an interface for the information processing apparatus 1 to present image information to the user, and is realized by a known liquid crystal panel, organic EL (Electroluminescence), or the like. The display unit 13 also has a touch sensor and functions as an input interface for the user to give instructions to the information processing apparatus 1 .

The communication unit 14 is a communication interface for the information processing device 1 to exchange information with external devices such as the content server 2 and the model server 3, and is realized by a known mobile phone module or Wi-Fi (registered trademark) module. ing. Therefore, the communication unit 14 functions as a transmitting unit for each unit constituting the control unit 11 to transmit information to an external device, and also functions as a receiving unit for receiving information from an external device.

The specific information acquisition unit 110 acquires specific information for specifying data including a city object V, which is a three-dimensional model of a structure in the real world. Although not limited, the city object V may be included as part of the VPS (Visual Positioning Service) data, or may be included in a form associated with the VPS data. Since VPS technology is a known technology, detailed description is omitted, but VPS data is data including three-dimensional data of existing structures such as buildings (for example, point cloud data representing the surface of a city). .

The matching unit 111 superimposes the city object V included in the data specified by the specific information acquired by the specific information acquiring unit 110 on the captured image being captured by the imaging unit 12 . The display control unit 112 causes the display unit 13 to display the captured image on which the city object V is superimposed. The model server 3 holds data including the city object V, and the matching unit 111 obtains the data from the model server 3 by inquiring the model server 3 using the specific information. Alternatively, if the data is already stored in the storage unit 10 , the matching unit 111 may read the data from the storage unit 10 to obtain the data.

The object management unit 113 selects an AR object (that is, a selection object A) to be installed in a space specified by a three-dimensional coordinate system that defines a city object V, which is a three-dimensional model, and The setting of the installation position and installation orientation of the selected object A is accepted.

The display control unit 112 causes the display unit 13 to display the selected object A at the installation position accepted by the object management unit 113 in the installation direction. This allows the user to grasp the relationship between the selected object A and the city object V without actually visiting an existing city.

FIG. 3 is a diagram schematically showing an example of a specific information specifying screen provided by the information processing device 1 according to the embodiment. Specifically, FIG. 3 shows an example in which the information processing device 1 is a smart phone and the display unit 13 is the screen of the information processing device 1 . As shown in FIG. 3, by touching the screen of the information processing device 1, the user searches for specific information for specifying the city object V using the place name, nearest station, or postal code, or searches for specific information on the map. It is possible to instruct the information processing apparatus 1 by directly specifying with . That is, the specific information acquisition unit 110 acquires from the user, as specific information, information indicating the position of the structure modeled by the city object V in the real world. As a result, as long as the model server 3 holds the data, the user can easily grasp the situation of the structure at home or the like without going to the vicinity of the structure in the real world.

FIG. 4 is a diagram for explaining matching between the real plane object R1 and the city object V. FIG. The matching unit 111 according to the embodiment detects a plane specified by two axes in the three-dimensional coordinate system C that defines the city object V as a planar object included in the captured image captured by the display unit 13. By matching with a certain real plane object R1, the captured image and the city object V are superimposed.

In FIG. 4, the three-dimensional coordinate system composed of the x-axis, y-axis, and z-axis defining the city object V is indicated by C, and the height direction is defined by the z-axis, x-axis, and y-axis. is the ground plane. Further, in the example shown in FIG. 4, the real plane object R1 included in the captured image captured by the display unit 13 is a desk top. The control unit 11 can superimpose the captured image and the city object V by matching the real plane object R1 with the plane defined by the x-axis and the y-axis. As a result, the user can observe an image as if a city diorama appeared on the desk.

FIG. 5 is a diagram schematically showing an example of a selection screen for selection object A provided by information processing apparatus 1 according to the embodiment. Similar to the diagram shown in FIG. 3 , FIG. 5 shows an example in which the information processing device 1 is a smart phone and the display unit 13 is the screen of the information processing device 1 .

FIG. 5 shows an example in which a penguin AR object is displayed as the selection object A. In FIG. Candidates for the selection object A are stored in the storage unit 10 , and the user can switch to another selection object A by tapping the display unit 13 .

Selection object A is modeled as a three-dimensional object. Therefore, as shown in FIG. 5, the user can observe the selected object from any angle including the front, back, and side of the selected object. Since the city object V is also modeled as a three-dimensional object, after superimposing the city object V and the selected object A, the user can observe the relationship between the city object V and the selected object A from any angle. can.

While understanding the relationship between the city object V and the selected object A, the user changes the placement position and placement direction of the selected object A, and finally determines. When the user determines the placement position and placement orientation of the selection object A, the communication unit 14 transmits object information including information specifying the selection object A and information indicating settings of the installation position and installation orientation of the selection object A. , to the content server 2 .

The information processing device 1 can communicate with the content server 2 from anywhere on the condition that the network N can be connected. Therefore, when the user of the information processing device 1 carries the information processing device 1, goes to the location of the structure that is the model of the city object V, and takes an image with the imaging unit 12, the display unit 13 displays the selected object as the captured image. An image with A superimposed is displayed. In this manner, the communication unit 14 transmits the object information to the content server 2, so that the user can share the set selection object A with other users at the site. Note that the matching unit 111 of the information processing device 1 can estimate the position and orientation of the imaging unit 12 from the captured image by using known VPS technology.

<Processing Flow of Information Processing Method Executed by Information Processing Apparatus 1>
FIG. 6 is a flowchart for explaining the flow of information processing executed by the information processing apparatus 1 according to the embodiment. The processing in this flowchart starts, for example, when the information processing apparatus 1 is activated.

The specific information acquisition unit 110 acquires from the user specific information for specifying data including the city object V (S2). The matching unit 111 receives and acquires the data specified by the specifying information from the model server 3 (S4).

The matching unit 111 matches the captured image being captured by the imaging unit 12 with the city object V, which is a three-dimensional model (S6). The display control unit 112 causes the display unit 13 to display a superimposed image in which the city object V is superimposed on the captured image (S8).

The object management unit 113 receives from the user a selection of a selection object A to be installed in the space specified by the three-dimensional coordinate system C that defines the city object V (S10). Further, the object management unit 113 receives from the user the setting of the installation position and installation orientation of the selection object A in the three-dimensional coordinate system C (S12).

The display control unit 112 causes the display unit 13 to display the selected selection object A at the set installation position in the installation orientation (S14). The communication unit 14 transmits object information including information specifying the selection object A and information indicating the setting of the installation position and installation orientation of the selection object A to the content server 2 (S16). When the communication unit 14 transmits the object information to the content server 2, the processing in this flowchart ends.

<Effects of Information Processing Apparatus 1 According to Embodiment>
As described above, according to the information processing apparatus 1 according to the embodiment, it is possible to provide a technique for grasping the relationship between the selection object A and the environment in which the selection object A is arranged.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed and integrated in arbitrary units. In addition, new embodiments resulting from arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment caused by the combination has the effect of the original embodiment.

1 Information processing apparatus 10 Storage unit 11 Control unit 110 Specific information acquisition unit 111 Matching unit 112 Display control unit 113 Object management unit 12 Imaging unit 13 Display unit 14 Communication unit 2 Contents server 3 Model server N Network S Display system

Claims (6)

a specific information acquisition unit that acquires specific information for specifying data including a three-dimensional model of a structure in the real world;
a matching unit that superimposes the three-dimensional model included in the data specified by the specific information on the captured image being captured by the imaging unit;
a display control unit that causes a display unit to display the captured image on which the three-dimensional model is superimposed;
Selection of an AR (Augmented Reality) object to be installed in a space specified by a three-dimensional coordinate system that defines the three-dimensional model, setting of the installation position and installation orientation of the AR object in the three-dimensional coordinate system; an object management unit that accepts
The display control unit causes the display unit to display the selected AR object at the installation position in the installation orientation.
Information processing equipment. further comprising a communication unit that transmits object information including information specifying the selected AR object and information indicating the setting of the installation position and installation orientation of the AR object to the content server;
The information processing device according to claim 1 . The specific information acquisition unit acquires information indicating a position where the structure exists in the real world from the user as the specific information.
The information processing apparatus according to claim 1 or 2. The matching unit matches a plane specified by two axes in a three-dimensional coordinate system that defines the three-dimensional model with a planar object included in the captured image, thereby matching the captured image and the three-dimensional model. superimposed with the model,
The information processing apparatus according to any one of claims 1 to 3. the processor
obtaining identification information for identifying data containing a three-dimensional model of a real-world structure;
a step of superimposing the three-dimensional model included in the data specified by the specific information on the captured image being captured by the imaging unit;
a step of displaying the captured image on which the three-dimensional model is superimposed on a display unit;
receiving selection of an AR object to be installed in a space specified by a three-dimensional coordinate system that defines the three-dimensional model, and setting of an installation position and an installation orientation of the AR object in the three-dimensional coordinate system; ,
causing the display unit to display the selected AR object at the installation position of the AR object in the installation orientation;
Information processing methods. to the computer,
the ability to obtain specific information for identifying data containing a three-dimensional model of a real-world structure;
a function of superimposing the three-dimensional model included in the data specified by the specific information on the captured image being captured by the imaging unit;
a function of displaying the captured image on which the three-dimensional model is superimposed on a display unit;
a function of accepting the selection of an AR object to be installed in a space specified by a three-dimensional coordinate system that defines the three-dimensional model, and the setting of the installation position and installation orientation of the AR object in the three-dimensional coordinate system; ,
a function of causing the display unit to display the selected AR object at the installation position of the AR object in the installation orientation;
program.

Images