JP7354186B2 - Display control device, display control method, and display control program - Google Patents

Description

The present invention relates to a display control device, a display control method, and a display control program.

In recent years, augmented reality (AR) has become popular as content provided via the Internet. In such AR, for example, an AR object viewed from a virtual viewpoint is displayed superimposed on a captured image (see, for example, Patent Document 1).

JP2020-95602A

However, with the conventional technology, there is room for improvement, for example, in terms of improving the sense of presence in AR content.

The present invention has been made in view of the above, and an object of the present invention is to provide a display control device, a display control method, and a display control program that can improve the sense of presence in AR content.

In order to solve the above-mentioned problems and achieve the purpose, a display control device according to the present invention includes an estimation section, a setting section, and a display control section. The estimating unit estimates a real object size that is a size of a real object that exists in the surrounding area. The setting unit superimposes display on the captured image of the surroundings based on the real object size estimated by the estimation unit and a virtual object size that is a size set for the virtual object in the multi-view image. A display size of the virtual object is set. The display control section displays the virtual object on the captured image at the display size set by the setting section.

According to the present invention, it is possible to improve the sense of presence in AR content.

FIG. 1 is a diagram showing an overview of a display control system according to an embodiment. FIG. 2 is a block diagram illustrating a configuration example of an information processing apparatus according to an embodiment. FIG. 3 is a diagram illustrating an example of information stored in the multi-view video database according to the embodiment. FIG. 4 is a block diagram illustrating a configuration example of a display control device according to an embodiment. FIG. 5 is a diagram illustrating an example of information stored in the size information storage unit according to the embodiment. FIG. 6 is a schematic diagram showing the correlation between the real object size and the display size of a virtual object according to the embodiment. FIG. 7 is a schematic diagram showing an example of a display surface according to the embodiment. FIG. 8 is a schematic diagram showing the relationship between a virtual object and a sound generation source according to the embodiment. FIG. 9 is a flowchart illustrating an example of a processing procedure executed by the information processing apparatus according to the embodiment. FIG. 10 is a flowchart illustrating an example of a processing procedure executed by the display control device according to the embodiment. FIG. 11 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device according to the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Below, a display control device, a display control method, and a mode for implementing a display control program (hereinafter referred to as an "embodiment") according to the present application will be described in detail with reference to the drawings. Note that the display control device, display control method, and display control program according to the present application are not limited to this embodiment.

[Embodiment]
[1. Display control processing]
First, an example of display control processing according to the embodiment will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of display control processing according to an embodiment.

As shown in FIG. 1, the display system S according to the embodiment includes, for example, an information processing device 10 and a display control device 50. The information processing device 10 is an information processing device that distributes various AR (Augmented Reality) contents to each display control device 50. The information processing device 10 is realized by a server device or a cloud system.

As shown in FIG. 1, the information processing device 10 has a multi-view video database, and provides multi-view videos registered in the multi-view video database as AR content 100. Here, the multi-view video is a video rendered by rendering a three-dimensional model from a plurality of predetermined viewpoints, or a video shot using a plurality of photographic equipment installed around an actual subject. Furthermore, each image forming the multi-view video becomes a multi-view image.

The display control device 50 is, for example, a variety of terminal devices such as a smartphone or an HMD (Head Mount Display) having a camera, lidar, etc., and allows the user U to view AR content 100 regarding multi-view video distributed from the information processing device 10. Provide possible.

For example, the display control device 50 provides AR content 100 in which a virtual object Ov of a multi-view video distributed from the information processing device 10 is superimposed on a captured image G of the surroundings. That is, the display control device 50 provides so-called video see-through type AR content 100. Note that the AR content 100 provided by the display system S is not limited to a video see-through type, but may be an optical see-through type.

By the way, for example, when providing AR content of multi-view video, there is a possibility that the size of a virtual object in the multi-view video may deviate from the actual size. To give a specific example, for example, when displaying a person as a virtual object as AR content, if you try to superimpose the AR content whose size setting does not correspond to the real world on the captured image G, the screen It may be displayed too large or too small to protrude from the screen.

These discrepancies in display size can become a problem in providing AR content that is closer to reality, that is, has a high sense of presence.

Therefore, the display control device 50 according to the embodiment estimates the real object size indicating the size of the real objects Or existing in the surroundings, and adjusts the display size of the virtual object Ov in the AR content 100 based on the estimated real object size. I decided to control it.

Specifically, as shown in FIG. 1, for example, the display control device 50 receives distribution of multi-view video from the information processing device 10 (step S01). For example, the display control device 50 displays the AR content 100 by performing so-called streaming playback based on the distributed multi-view video.

At this time, the display control device 50 receives, as a multi-view video, information regarding a multi-view image of the virtual object viewed from an arbitrary viewpoint position, information regarding the virtual object size set for each virtual object Ov in the multi-view video, etc. .

Subsequently, the display control device 50 executes a process of estimating the real object size, which is the size of the real object Or (step S02). The display control device 50 estimates the actual object size by, for example, performing image recognition on a captured image using a camera.

More specifically, since the approximate size of a predetermined real object Or such as furniture is determined depending on the type, etc., the real object Or serving as a reference such as furniture is recognized by image recognition using an image captured by a camera, and The real object size of the recognized real object Or is estimated.

Further, the display control device 50 calculates the distance to the real object Or, etc., based on the distance measurement result of a distance measurement sensor such as Lidar, for example. Based on the estimated real object size and the distance from the display control device 50 to the real object Or, real space information such as the overall size and arrangement of the space in which the real object Or actually exists (hereinafter referred to as real space) is then obtained. Estimate.

FIG. 1 illustrates a case where the real object Or is a curtain, and the height h and width w of the curtain are estimated as the real object. Subsequently, the display control device 50 performs display control processing to control the display size of the virtual object based on the real object size estimated in step S2 (step S03).

For example, in the display control process of step S3, the display control device 50 controls the display size of the virtual object Ov so that the real object size of the real object Or matches the virtual object size of the virtual object Ov.

That is, the display control device 50 controls the display size of the virtual object Ov in the AR content 100 so that the virtual object Ov has the actual size (actual size) when the virtual object Ov is actually placed in real space. do.

Thereby, the display control device 50 according to the embodiment can provide the AR content 100 with higher reality, so that the sense of presence in the AR content 100 can be improved.

[2-1. Configuration example of information processing device]
Next, a configuration example of the information processing device 10 will be described using FIG. 2. FIG. 2 is a block diagram showing a configuration example of the information processing device 10. As shown in FIG. As shown in FIG. 2, the information processing device 10 includes a communication section 20, a storage section 30, and a control section 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card). The communication unit 20 connects networks such as various wireless communication networks such as 4G (Generation), 5G, LTE (Long Term Evolution), Wifi (registered trademark), or wireless LAN (Local Area Network), or various wired communication networks. Information is sent and received to and from external devices via the external device.

The storage unit 30 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. Furthermore, the storage unit 30 has a multi-view video database 31.

The multi-view video database 31 is a database that stores multi-view videos. FIG. 3 is a diagram showing an example of information stored in the multi-view video database 31. As shown in FIG. 3, the multi-view video database 31 stores information on items such as "video ID", "image group", "size information", "audio", "sound source coordinates", etc. in association with each other.

"Video ID" is an identifier for identifying each multi-view video, and "image group" is a group of images to be provided as multi-view video, and is a still image or a moving image shot simultaneously from multiple viewpoints. This data is a compilation of the following. Note that the image group may be replaced with a 3D model (for example, three-dimensional graphic data).

"Size information" is information regarding the actual size of an object photographed as a multi-view video. "Audio" indicates audio data that is played along with the multi-view video, and "sound source coordinates" is information regarding the coordinates of the sound source of the audio data in the multi-view video.

When providing a singing video of an artist as the AR content 100, the audio is the sound source of the artist singing, and the sound source coordinates are the artist's head (for example, the mouth).

Returning to the explanation of FIG. 2, the control section 40 will be explained. The control unit 40 is, for example, a controller, and is stored in a storage device inside the information processing device 10 by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processing Unit), etc. This is realized by executing various programs using the RAM as a work area.

As shown in FIG. 2, the control section 40 includes a selection section 41 and a distribution section 42. The selection unit 41 selects a corresponding multi-view video from the multi-view video database 31 based on the multi-view video distribution request received from each display control device 50 .

For example, the distribution request includes information regarding the above video ID, and the selection unit 41 selects a multi-view video that matches the video ID included in the distribution request from the multi-view video database 31.

Further, for example, during distribution of multi-view video, the selection unit 41 receives user's point of interest information on the AR content 100 from each display control device 50, and performs viewpoint switching processing on the multi-view video based on the point of interest information. .

For example, the gaze point information is information about the imaging range and imaging direction of the captured image G in real space, and is information about the real space displayed as the AR content 100. That is, it is information regarding the user's gaze point in the AR content 100. For example, when the user changes the position of the display control device 50 (for example, an outward-facing camera 61 to be described later), the point of interest changes, and information regarding the changed direction, amount of change, and the like becomes point of interest information.

The selection unit 41 performs viewpoint switching processing by selecting a viewpoint position for a multi-view video based on the gaze point information. As a result, a multi-view image of the virtual object Ov viewed from the selected viewpoint position is selected. Thereafter, the selection unit 41 performs a process of changing the viewpoint position as needed to follow changes in the gaze point information.

The distribution unit 42 distributes the multi-view video selected by the selection unit 41 to each display control device 50. Note that the multi-view video distributed by the distribution unit 42 is a temporally continuous multi-view image, and the distribution unit 42 continuously transmits multi-view images viewed from the viewpoint position set based on the point of interest information. To deliver. Further, the distribution unit 42 distributes information regarding audio and the like together with the multi-view video to the display control device 50.

[2-2. Configuration example of display control device]
Next, a configuration example of the display control device 50 according to the embodiment will be described using FIG. 4. FIG. 4 is a block diagram showing a configuration example of the display control device 50. As shown in FIG. As shown in FIG. 4, the display control device 50 includes a communication section 60, an outward camera 61, an inward camera 62, a sensor group 63, a display section 64, an audio output section 65, a storage section 70, and a control section 80. Note that the outward camera 61, the inward camera 62, the sensor group 63, the display section 64, and the audio output section 65 may be separate devices from the display control device 50.

The communication unit 60 is realized by, for example, a NIC (Network Interface Card). The communication unit 60 connects networks such as various wireless communication networks such as 4G (Generation), 5G, LTE (Long Term Evolution), WiFi (registered trademark), or wireless LAN (Local Area Network), or various wired communication networks. Information is sent and received to and from external devices via the external device.

The outward camera 61 is a camera that images the outside of the display control unit 50 (for example, the side opposite to the display surface of the display unit 64), and images the real space. For example, a captured image captured by the outward camera 61 is used as a background image of the virtual object Ov in the AR content 100. The inward camera 61 is a camera that captures an image of the inside of the display control unit 50 (for example, the display surface side of the display unit 64), and captures an image of the user U viewing the AR content 100.

The sensor group 63 includes various sensors for detecting, for example, the attitude of the display control device 50 and the distance to the real object Or existing in real space. For example, the sensor group 63 includes a gyro sensor, lidar, and the like.

The display unit 64 is, for example, a touch panel display, and displays various images such as the AR content 100 and receives various user operations. The audio output unit 65 is, for example, a surround speaker or a stereo speaker, and outputs various sounds that flow together with the AR content 100.

The storage unit 70 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 70 also includes a map information storage unit 71 and a size information storage unit 72.

The map information storage unit 71 stores map information of real space. For example, the map information is an environmental map showing the surrounding environment of the display control unit 50, and is created based on processing results by the self-position estimating unit 81 and the size estimating unit 82, which will be described later.

The size information storage unit 72 stores information regarding the real object size of the real object Or. FIG. 5 is a diagram showing an example of information stored in the size information storage section 72. As shown in FIG. 5, the size information storage unit 72 stores information on items such as "object ID," "feature amount," and "size information" in association with each other.

The "object ID" is an identifier for identifying the real object Or, and the "feature amount" indicates the feature amount of the real object Or. The feature amount here indicates, for example, the appearance feature of the real object Or.

"Size information" is information regarding the size of the corresponding real object Or, and indicates, for example, information regarding the actual size of the real object Or, that is, the life-size size. Note that the size information may be a catalog value or an actual measurement value. Alternatively, the value measured by the display control device 50 may be used as the actual measurement value.

Returning to the explanation of FIG. 4, the control section 80 will be explained. The control unit 80 is, for example, a controller, and is stored in a storage device inside the information processing device 10 by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processing Unit), etc. This is realized by executing various programs using the RAM as a work area.

As shown in FIG. 4, the control unit 80 includes a self-position estimation unit 81, a size estimation unit 82, a display screen setting unit 83, a display control unit 84, and an audio control unit 85. The self-position estimating unit 81 uses technology such as Slam (Simultaneous Localization and Mapping) to create an environmental map of the real space based on various information input from the outward facing camera 61 and the sensor group 63. The coordinates (for example, world coordinates) and orientation of the display control device 50 at are estimated.

Further, the self-position estimating unit 81 stores information regarding the created environmental map as map information in the map information storage unit 71, and also stores information regarding the estimated position and orientation, that is, the above-mentioned point of interest information, to the display surface setting unit 83. pass it on to Furthermore, while the AR content 100 is being provided, the gaze point information is also transmitted to the information processing device 10.

The size estimating unit 82 estimates a real object size that is the size of a real object Or existing around the real object Or. For example, the size estimation unit 82 recognizes the real object Or from the captured image taken by the outward camera 61 and estimates the real object size.

For example, the size estimating unit 82 compares the feature amount of each real object Or stored in the size information storage unit 72 with the feature amount of the object included in the captured image, so that the size estimation unit 82 matches the feature amount of both objects. The type of the surrounding real object Or is recognized from the ratio.

When the size estimation unit 82 recognizes the type of the real object Or, it estimates the real object size of the real object Or based on the size information stored in the size information storage unit 72. Further, the size estimation unit 82 estimates the distance from the outward camera 61 to the real object Or, for example, based on the detection results of the sensor group 63 (for example, Lidar).

For example, the size estimation unit 82 estimates the magnification (scale) of the real object Or in the captured image based on the estimated real object size and the distance to the corresponding real object Or.

The display surface setting unit 83 superimposes the image on the surrounding image based on the real object size estimated by the size estimating unit 82 and the virtual object size which is the size set for the virtual object Ov in the multi-view image. Set the display size of the virtual object Ov to be displayed. For example, the display surface setting unit 83 inquires of the information processing device 10 about size information of the virtual part object Ov provided as the AR content 100, and sets the display size based on the inquired size information.

FIG. 6 is a schematic diagram showing the correlation between the real object size and the display size of the virtual object Ov. In the example shown in FIG. 6, a case will be described in which a virtual object Ov to be displayed superimposed on the captured image G is placed in front of the real object Or shown in the captured image G at a distance d from the real object Or.

In this case, the height h2 at which the virtual object Ov is displayed in the AR content 100 is set based on the height h1 of the real object Or. That is, the height h2, which is the display size of the virtual object Ov, is set based on the positional relationship between the real object Or and the virtual object Ov to be displayed, and the ratio of their actual sizes.

In other words, the display surface setting unit 83 calculates the height h2 so that the virtual object Ov is displayed in its original size based on the height h1 of the real object Or, and calculates the height h2 of the virtual object Ov in the multi-view video. The display size of the virtual object Ov is set so that the height becomes the height h2.

Further, the display surface setting unit 83 initializes a virtual display surface (hereinafter simply referred to as a display surface) on which the virtual object Ov is displayed from the captured image, based on the result of setting the display size of the virtual object Ov.

FIG. 7 is a schematic diagram showing an example of a display surface. In the example of FIG. 7, a case will be described in which two virtual objects Ov1 and Ov2 are displayed.

For example, based on the imaging space of the current captured image G, the display surface setting unit 83 sets a space that can contain both the virtual objects Ov1 and Ov2 as the display area A in which the virtual objects Ov1 and Ov2 are displayed, and sets the display area A The display surface Ap is initialized by setting the bottom surface of the display surface Ap as the display surface Ap.

In other words, the display surface setting unit 83 does not set a space that cannot be included in the virtual objects Ov1 and Ov2 as the display area A. Thereby, for example, in the AR content 100, it is possible to avoid in advance an erroneous display in which the virtual object Ov is displayed penetrating through the wall or ceiling of the real space.

Thereafter, the display surface setting section 83 writes the coordinate information of the set display area A and display surface Ap into the map information storage section 71. Note that, for example, the display surface Ap is the floor in real space, but it may also be a part of the user's body (for example, the palm of the hand) depending on the type of the virtual object Ov. Furthermore, the display area A itself may be made to float above the actual floor. Further, the display surface setting unit 83 may independently set the display area A and the display surface Ap for the virtual objects Ov1 and Ov2. Furthermore, for example, there may be content in which the virtual object Ov itself moves within the AR content 100, so the display surface setting unit 83 sets the display area A and the display surface Ap so as to include the moving range of the virtual object Ov. You can do it like this.

Thereafter, the display screen setting unit 83 updates the display size according to the imaging position of the captured image G. For example, the display surface setting unit 83 recognizes the imaging position of the captured image G (for example, the position of the outward camera 61) based on the gaze point information received from the self-position estimation unit 81. Then, the display screen setting unit 83 sets the display size based on the positional relationship between the imaging position and the display screen Ap. Specifically, the display surface setting unit 83 sets the display size so that the closer the imaging position and the display surface Ap get, the larger the display size becomes, and the farther away the imaging position and the display surface Ap, the smaller the display size becomes. .

That is, the display surface setting unit 83 updates the display size of the virtual object Ov as needed so that it becomes the actual size that appears in the captured image G, assuming that the virtual object Ov exists on the display surface Ap.

Returning to the explanation of FIG. 4, the display control section 84 will be explained. The display control unit 84 displays the virtual object Ov on the captured image G at the display size set by the display surface setting unit 83. First, the display control unit 83 requests the information processing device 10 to distribute multi-view video, and starts acquiring the multi-view video from the information processing device 10 .

For example, the display size of the virtual object Ov of each multi-view image constituting the multi-view video is set by the display surface setting unit 83. The display control unit 84 generates the AR content 100 by superimposing and displaying the virtual object Ov with the set display size on the captured image G obtained from the outward facing camera 61. Then, the display control unit 84 displays the AR content 100 on the display unit 64 every time it is generated.

By the way, in AR content, for example, there may be a desire to display only some of the virtual objects Ov in a multi-view video, or a desire to change the arrangement or orientation of the virtual objects Ov.

Therefore, for example, the display control unit 84 may display only some of the virtual objects Ov selected by the user's operation, or may change the arrangement or orientation of the virtual objects Ov.

To give a more specific example, for example, if each member of an idol group is an independent virtual object Ov, it is possible to display only the virtual objects Ov corresponding to some members, or change the arrangement of the virtual objects Ov. By doing so, you can change the positions of members.

For example, the display control unit 84 receives the above information regarding the user operation from the display unit 64, and requests the information processing device 10 to distribute multi-view images of some virtual objects Ov based on the received information regarding the user operation. do. Thereby, it is possible to provide AR content 100 in which only some virtual objects Ov are displayed in a superimposed manner. Note that the display control unit 84 may display only some of the virtual objects Ov by performing a predetermined masking process on the virtual objects Ov that are not displayed.

Furthermore, the display control unit 84 can provide the AR content 100 in which the arrangement and orientation of the virtual object Ov have been changed by changing the arrangement and orientation of the corresponding virtual object Ov based on the information regarding the user operation. .

In addition, the display control unit 84 may, for example, add a virtual object Ov to be displayed as the AR content 100 based on a user operation. For example, it is possible to provide AR content 100 that combines virtual objects Ov corresponding to members selected by the user from a plurality of multi-view videos taken at different live performances.

That is, the AR content 100 of a member selected by the user from a plurality of live videos may be provided. At this time, for example, the selected virtual objects Ov may be the same person or different persons.

Furthermore, during playback of the AR content 100, the display control unit 84 may decide to pause only some of the virtual objects Ov or change the playback speed based on user operations. That is, the display control unit 84 may set a plurality of time axes for each virtual object Ov within the AR content 100. Furthermore, the display control unit 84 may accept changes in the size of the virtual object Ov.

The audio control unit 85 controls audio output based on the positional relationship between the imaging position of the captured image G and the audio source set in the multi-view image. FIG. 8 is a schematic diagram showing the relationship between the virtual object Ov and the audio source. FIG. 8 shows three-axis orthogonal coordinates, which correspond to a world coordinate system showing the positional relationship between the outward camera 61 and the virtual object Ov.

As shown in FIG. 8, for example, a sound source Ss is set at the head of the virtual object Ov. That is, in the example shown in FIG. 8, the audio control unit 85 controls audio output so that audio is generated from the virtual object Ov.

Here, for example, when the imaging position of the outward camera 61 changes, the position of the virtual object Ov in the AR content 100 changes, and the position of the sound source Ss changes accordingly.

Therefore, the audio control unit 85 performs processing to match the positions of the audio generation sources Ss in the AR content 100 by controlling the output of the audio output unit 65 based on these positional relationships. Specifically, for example, when the virtual object Ov is displayed on the right side in the captured image G, the audio output unit 65 is controlled so that the audio can be heard from the right side of the user, and the virtual object Ov is displayed on the front side in the captured image G. When displayed, the audio output section 65 is controlled so that the audio can be heard from the front side.

In this way, the audio control unit 85 can provide highly realistic AR content 100 by controlling the audio control in accordance with the change in the position of the audio source Ss. Note that although the above example shows a case where there is one sound generation source Ss, there may be a plurality of sound generation sources Ss. In this case, the audio control unit 85 performs audio control based on the positional relationship for each audio source Ss.

Further, for example, when there are a plurality of sound generation sources Ss, the audio control unit 85 may perform various adjustments such as the volume for each sound generation source Ss, or may integrate the sound generation sources Ss. Further, the audio control unit 85 may accept, for example, an operation to change the position of the audio source Ss. In this case, for example, the sound source Ss may be set at arbitrary coordinates different from the initially set sound source Ss.

Furthermore, the audio control unit 85 may delete or add audio based on user operations, for example. That is, arbitrary audio may be combined and output from a plurality of multi-view videos. In this case, for example, it is possible to provide AR content 100 that makes it appear as if the virtual object Ov of another singer is singing the voice of a different singer.

[3. Processing procedure]
Next, processing procedures executed by the information processing device 10 and the display control device 50 according to the embodiment will be described using FIGS. 9 and 10. First, the processing procedure executed by the information processing device 10 will be described using FIG. 9. FIG. 9 is a flowchart showing a processing procedure executed by the information processing device 10. Note that the processing procedure described below is repeatedly executed by the control unit 40 of the information processing device 10 every time a distribution request is obtained.

As shown in FIG. 9, the information processing device 10 first receives a distribution request (step S101). Subsequently, the information processing device 10 distributes the multi-view video based on the received distribution request (step S102), and ends the process.

Note that when distributing a multi-view video, the information processing device 10 selects a multi-view image of the virtual object Ov whose viewpoint position has been changed based on the point of interest information, and distributes the selected multi-view image.

Next, the processing procedure executed by the display control device 50 will be described using FIG. 10. FIG. 10 is a flowchart illustrating an example of a processing procedure executed by the display control device 50. Note that the processing procedure described below is executed by the control unit 80 of the display control device 50 each time the AR content 100 is provided.

As shown in FIG. 10, the display control device 50 first estimates the real object size of the real object Or existing in the surroundings, based on, for example, a captured image of the surroundings (step S201).

Subsequently, the display control device 50 determines the virtual object size when displaying the virtual object Ov based on the estimated real object size (step S202). Subsequently, the display control device 50 sets the display area A of the virtual object Ov based on the determined virtual object size (step S203).

After that, the display control device 50 acquires the multi-view video from the information processing device 10 (step S204), displays the AR content 100 in which the acquired multi-view video is superimposed on the captured image G (step S205), and performs the processing. finish.

[4. Modified example]
The display control system S described above may be implemented in various different forms other than the embodiments described above. Therefore, other embodiments of the display control system S will be described below.

For example, the display control device 50 may be configured separately from the cameras (outward camera 61 and inward camera 62). Further, the content provided by the display control device 50 is not limited to the AR content 100, but may also provide VR (Virtual Reality) content in which real space is converted into animation or the like based on the captured image G, for example.

[5. effect〕
As described above, the display control device 50 according to the embodiment includes the size estimating unit 82 (an example of an estimating unit) that estimates the real object size that is the size of the real object Or existing in the surroundings, and the size estimating unit 82 Based on the estimated real object size and the virtual object size, which is the size set for the virtual object Ov in the multi-view image, the display size of the virtual object Ov to be superimposed and displayed on the captured image G of the surroundings is set. It includes a display screen setting section 83 (an example of a setting section) and a display control section 84 that displays the virtual object on the captured image G at the display size set by the display screen setting section 83.

Therefore, according to the display control device 50 according to the embodiment, since the display size of the virtual object Ov can be displayed appropriately (actual size), the sense of realism in the AR content 100 can be improved.

Further, in the display control device 50 according to the embodiment, the display screen setting unit 83 sets the display size according to the imaging position of the captured image G. Thereby, the display control device 50 according to the embodiment can set the virtual object Ov to an appropriate display size according to the viewpoint position in the real world in the AR content 100.

In the display control device 50 according to the embodiment, the display control unit 84 superimposes and displays some of the virtual objects Ov selected from among the plurality of virtual objects Ov in the multi-view image on the captured image G, and selects The virtual object Ov that has not been displayed is not displayed in a superimposed manner on the captured image G. Therefore, according to the display control device 50 according to the embodiment, the user can easily customize the AR content 100.

Furthermore, in the display control device 50 according to the embodiment, the display control unit 84 combines virtual objects Ov selected from a plurality of multi-view images and displays the virtual objects Ov in a superimposed manner on the captured image G. Therefore, according to the display control device 50 according to the embodiment, it is possible to easily generate AR content 100 that is a combination of a plurality of contents.

In the display control device 50 according to the embodiment, the display control unit 84 changes the arrangement of some selected virtual objects Ov among the plurality of virtual objects Ov in the multi-view image and superimposes them on the captured image G. indicate. Therefore, according to the display control device 50 according to the embodiment, the user can easily customize the AR content 100.

The display control device 50 according to the embodiment also includes an audio control unit 85 that controls audio output based on the positional relationship between the imaging position of the captured image and the audio source set in the multi-view image. Therefore, according to the display control device 50 according to the embodiment, it is possible to match the sound generation source Ss and the virtual object Ov, and therefore it is possible to provide the AR content 100 with a higher sense of realism.

In the display control device 50 according to the embodiment, the audio control unit 85 outputs audio associated with a plurality of multi-view images within the same image. Therefore, according to the display control device 50 according to the embodiment, since separately recorded sounds can be provided as the same AR content 100, the user can easily customize the AR content 100.

Further, the display control method according to the embodiment is a display control method executed by a computer, and includes an estimation step of estimating a real object size that is the size of a real object Or existing in the surroundings, and a real object size estimated by the estimation step. a setting step of setting the display size of the virtual object Ov to be superimposed and displayed on the captured image G of the surroundings based on the object size and the virtual object size that is the size set for the virtual object Ov in the multi-view image; and a display control step of displaying the virtual object Ov on the captured image G at the display size set in the setting step. Therefore, according to the display control method according to the embodiment, the virtual object Ov can be displayed at an appropriate display size (actual size), so that the sense of presence in the AR content 100 can be improved.

Further, the display control program according to the embodiment includes an estimation procedure for estimating a real object size that is the size of a real object Or existing in the surroundings, a real object size estimated by the estimation procedure, and a virtual object in a multi-view image. A setting procedure for setting the display size of the virtual object Ov to be superimposed and displayed on a captured image G of the surroundings based on the virtual object size, which is the size set in Ov; The computer is caused to execute a display control procedure for displaying Ov on the captured image G. Therefore, according to the display control program according to the embodiment, since the display size of the virtual object Ov can be displayed appropriately (actual size), the sense of presence in the AR content 100 can be improved.

[6. Hardware configuration]
Further, the display control device 50 according to the embodiments described above is realized by, for example, a computer 1000 having a configuration as shown in FIG. 11. FIG. 11 is a hardware configuration diagram showing an example of a computer that implements the functions of the display control device 50 according to the embodiment. Computer 1000 has CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I/F) 1500, input/output interface (I/F) 1600, and media interface (I/F) 1700.

CPU 1100 operates based on a program stored in ROM 1300 or HDD 1400, and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, programs depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. Communication interface 1500 receives data from other devices via network (communication network) N and sends it to CPU 1100, and sends data generated by CPU 1100 to other devices via network N.

The CPU 1100 connects output devices such as displays and printers, and input devices such as keyboards and mice (in FIG. 10, output devices and input devices are collectively referred to as "input/output devices") via an input/output interface 1600. ). CPU 1100 obtains data from an input device via input/output interface 1600. Further, CPU 1100 outputs the generated data to an output device via input/output interface 1600.

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200. CPU 1100 loads this program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the display control device 50 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 120 by executing a program loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired from another device via the network N.

Some of the embodiments of the present application have been described above in detail based on the drawings, but these are merely examples, and various modifications and variations may be made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure section of the invention. It is possible to carry out the invention in other forms with modifications.

[7. others〕
Furthermore, among the processes described in the above embodiments and modified examples, all or part of the processes described as being performed automatically can be performed manually, or may be described as being performed manually. All or part of this processing can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

Furthermore, the above-described embodiments and modifications can be combined as appropriate within a range that does not conflict with the processing contents.

Further, the above-mentioned "section, module, unit" can be read as "means", "circuit", etc. For example, the self-position estimating section 81 can be read as a self-position estimating section means or a self-position estimating section circuit.

10 Information processing device 20 Communication unit 30 Storage unit 31 Multi-view video database 40 Control unit 41 Selection unit 42 Distribution unit 50 Display control unit 70 Storage unit 71 Map information storage unit 72 Size information storage unit 80 Control unit 81 Self-position estimation unit 82 Size estimator (an example of an estimator)
83 Display screen setting section (an example of a setting section)
84 Display control unit 85 Audio control unit 100 AR content G Captured image Or Real object Ov Virtual object

Claims (8)

an estimation unit that estimates a real object size that is the size of a real object that exists in the surroundings;
The surrounding area is imaged based on the real object size estimated by the estimation unit and a virtual object size that is a size set for a virtual object displayed as a multi-view image obtained by photographing the real object from multiple viewpoints. a setting unit that sets a display size of the virtual object to be displayed superimposed on the captured image;
a display control unit that displays the virtual object in the captured image at the display size set by the setting unit;
The setting section includes:
setting as a display area of the virtual object a space that includes a movement range in which the virtual object moves during playback and in which the virtual object does not collide with any of the real objects during playback ;
The display control section includes:
Displaying the virtual object viewed from a viewpoint position that corresponds to a change in the imaging position of the captured image, and displaying the virtual object at the display size that corresponds to a change in the positional relationship between the virtual object and the real object. A display control device characterized by: The setting section includes:
The display control device according to claim 1, wherein the display size is set according to the imaging position of the captured image. The display control section includes:
Some of the virtual objects selected from among the plurality of virtual objects in the multi-view image are displayed in a superimposed manner on the captured image, and virtual objects that are not selected are not displayed in a superimposed manner on the captured image. The display control device according to claim 1 or 2. The display control section includes:
The display control device according to claim 1, wherein the virtual objects in the plurality of multi-view images are combined and displayed in a superimposed manner on the captured image. The display control section includes:
Any one of claims 1 to 4, characterized in that a selected part of the virtual objects in the multi-view image is rearranged and displayed in a superimposed manner on the captured image. The display control device described in . The display control section includes:
The display control device according to any one of claims 1 to 5, wherein changes in the display size, playback, and pause are accepted for each of the virtual objects. A display control method executed by a computer,
an estimation step of estimating a real object size that is the size of a real object existing in the surroundings;
The surrounding area is imaged based on the real object size estimated in the estimation step and a virtual object size that is a size set for a virtual object displayed as a multi-view image obtained by photographing the real object from multiple viewpoints. a setting step of setting a display size of the virtual object to be displayed superimposed on the captured image;
a display control step of displaying the virtual object in the captured image at the display size set by the setting step;
The setting step includes:
setting as a display area of the virtual object a space that includes a movement range in which the virtual object moves during playback and in which the virtual object does not collide with any of the real objects during playback ;
The display control step includes:
Displaying the virtual object viewed from a viewpoint position that corresponds to a change in the imaging position of the captured image, and displaying the virtual object at the display size that corresponds to a change in the positional relationship between the virtual object and the real object. A display control method characterized by: an estimation procedure for estimating a real object size that is the size of a real object existing in the surroundings;
The surrounding area is imaged based on the real object size estimated by the estimation procedure and a virtual object size that is a size set for a virtual object displayed as a multi-view image obtained by photographing the real object from multiple viewpoints. a setting procedure for setting a display size of the virtual object to be displayed superimposed on a captured image;
causing a computer to execute a display control procedure for displaying the virtual object in the captured image at the display size set by the setting procedure;
The setting procedure is as follows:
setting as a display area of the virtual object a space that includes a movement range in which the virtual object moves during playback and in which the virtual object does not collide with any of the real objects during playback ;
The display control procedure includes:
Displaying the virtual object viewed from a viewpoint position that corresponds to a change in the imaging position of the captured image, and displaying the virtual object at the display size that corresponds to a change in the positional relationship between the virtual object and the real object. A display control program featuring:

Images