JP7118217B1 - Display control device, display control method and display control program - Google Patents

Abstract

An object of the present invention is to provide AR content suitable for a viewing mode. A display control device according to the present application includes a display control unit that displays a superimposed image in which a multi-viewpoint image is superimposed on a captured image of the surroundings, and a viewing mode of the superimposed image displayed by the display control unit. and an adjusting unit that adjusts the color of the captured image or the multi-viewpoint image in the superimposed image according to. [Selection drawing] Fig. 1

Description

The present invention relates to a display control device, a display control method, and a display control program.

In recent years, augmented reality (AR) is becoming popular as content provided via the Internet. In such AR, for example, a planar image of three-dimensional data viewed from a virtual viewpoint is superimposed on a captured image (see, for example, Japanese Patent Application Laid-Open No. 2002-200013).

Japanese Patent Application Laid-Open No. 2020-95602

However, in the conventional technology, there is room for improvement in providing AR content suitable for the viewing mode.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a display control device, a display control method, and a display control program capable of providing AR content suitable for viewing modes.

In order to solve the above-described problems and achieve the object, a display control device according to the present invention includes a display control section and an adjustment section. The display control unit displays a superimposed image in which a multi-viewpoint image is superimposed on an imaged image of the surroundings. The adjustment unit adjusts the colors of the captured image or the multi-viewpoint image in the superimposed image according to the viewing mode of the superimposed image displayed by the display control unit.

According to the present invention, it is possible to provide AR content suitable for viewing modes.

FIG. 1 is a diagram illustrating an example of display control processing according to the embodiment. FIG. 2 is a block diagram illustrating a configuration example of the information processing apparatus according to the embodiment; FIG. 3 is a diagram illustrating an example of information stored in a multi-view video database according to the embodiment; FIG. 4 is a block diagram illustrating a configuration example of the display control device according to the embodiment; FIG. 5 is a diagram illustrating an example of processing by an adjustment unit according to the embodiment; 6 is a diagram illustrating an example of processing by an adjustment unit according to the embodiment; FIG. 7A and 7B are diagrams illustrating an example of an operation on a light emitter according to the embodiment; FIG. 8 is a flowchart illustrating an example of a processing procedure executed by the information processing apparatus according to the embodiment; FIG. 9 is a flowchart illustrating an example of a processing procedure executed by the display control device according to the embodiment; FIG. FIG. 10 is a hardware configuration diagram showing an example of a computer that implements the functions of the display control device according to the embodiment.

EMBODIMENT OF THE INVENTION Below, the form (it is hereafter described as "embodiment".) for implementing the display control apparatus, display control method, and display control program which concerns on this application is demonstrated in detail, referring drawings. Note that the display control device, the display control method, and the display control program according to the present application are not limited to this embodiment.

[Embodiment]
[1. Display control processing]
First, an example of display control processing according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of display control processing according to the embodiment.

As shown in FIG. 1, the display system S according to the embodiment has, for example, an information processing device 10 and a display control device 50 . The information processing device 10 is an information processing device that distributes various types of AR (Augmented Reality) content to each display control device 50 . The information processing device 10 is implemented by a server device or a cloud system.

As shown in FIG. 1 , the information processing device 10 has a multi-view video database, and provides multi-view video registered in the multi-view video database as AR content 100 . Here, the multi-viewpoint video is a video obtained by rendering a three-dimensional model from a plurality of predetermined viewpoints, or a video captured by installing a plurality of shooting devices around an actual subject. Also, each image constituting the multi-view video is a multi-view image.

The display control device 50 is, for example, various terminal devices such as a smartphone and an HMD (Head Mount Display), and provides the user with the AR content 100 distributed from the information processing device 10 so that the user can view it.

For example, the display control device 50 provides the AR content 100 in which the virtual object Ov of the multi-view video distributed from the information processing device 10 is superimposed on the captured image G of the surroundings. That is, the display control device 50 provides so-called video see-through type AR content 100 . Note that the AR content 100 provided by the display system S is not limited to the video see-through type, and may be the optical see-through type.

By the way, for example, in providing multi-view video AR content, there is room for improvement in providing AR content suitable for viewing modes. As an example of this, for example, when strong light such as the afternoon sun is irradiated in the real space, the captured image G reflects the light.

On the other hand, for example, the virtual object Ov is displayed independently of the influence of light in the real space. Therefore, in the AR content 100, there may be a difference in the amount of light between the virtual object Ov and its surroundings, which may give an uncomfortable feeling. be.

In the display control process according to the embodiment, focusing on such points, the color of the captured image G or the virtual object Ov is adjusted according to the viewing mode. Specifically, as shown in FIG. 1, the display control device 50 starts acquiring multi-viewpoint images from the information processing device 10 (step S01). Subsequently, the display control device 50 performs display control processing of the AR content 100 by playing back and displaying moving images of the multi-view video each time the multi-view video is acquired (step S02).

In the example shown in FIG. 1, the virtual object Ov is an idol group, and a case is shown in which a singing video of the idol group is superimposed on the captured image G as the virtual object Ov.

Subsequently, the display control device 50 performs adjustment processing for adjusting the color of the captured image G or the virtual object Ov based on the viewing mode (step S03). Here, color includes, for example, brightness and color temperature.

Also, the viewing mode here indicates the brightness of the surroundings, that is, the intensity of the light in the captured image G, and the like. That is, it indicates that the viewing mode differs depending on whether the captured image G reflects strong light or when the captured image G is captured in a dark place.

For example, the display control device 50 detects the light control state of the real space, such as the intensity of light in the captured image G and the irradiation range, by performing various image analyzes on the captured image G. FIG. Then, the display control device 50 determines whether color adjustment of the captured image G or the virtual object Ov is necessary based on the brightness of the virtual object Ov displayed in the AR content 100 .

As a more specific example, if a dark virtual object Ov is displayed in spite of the fact that the surroundings are bright, a color separation occurs between the virtual object Ov and its surroundings. On the other hand, the virtual object Ov can be appropriately blended into the captured image G by brightening the color of the captured image G or darkening the color of the virtual object Ov, for example.

In the example shown in FIG. 1, the light L of the real space is reflected on the back side of the virtual object Ov appearing in the AR content 100, and the amount of light differs between the virtual object Ov and its surroundings due to the light L. . Then, an example of adjusting the color in the display area of the light L in the captured image G will be shown. As a result, the AR content 100 is corrected so that the light L disappears. Note that the example of color adjustment is not limited to the above example, and the color of the virtual object Ov existing in the irradiation area of the light L is adjusted, and the AR content 100 is displayed as if the light L is applied to the virtual object Ov. You may decide to correct it.

Thus, in the display control process according to the embodiment, the color of the captured image G or the virtual object Ov is adjusted according to the viewing mode. Therefore, according to the display control process according to the embodiment, it is possible to provide the AR content 100 suitable for the viewing mode.

[2-1. Configuration example of information processing device]
Next, a configuration example of the information processing apparatus 10 will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration example of the information processing apparatus 10. As shown in FIG. As shown in FIG. 2 , the information processing device 10 includes a communication section 20 , a storage section 30 and a control section 40 .

The communication unit 20 is implemented by, for example, a NIC (Network Interface Card) or the like. The communication unit 20 uses networks such as various wireless communication networks such as 4G (Generation), 5G, LTE (Long Term Evolution), Wifi (registered trademark) or wireless LAN (Local Area Network), or various wired communication networks. Information is sent and received to and from an external device via the network.

The storage unit 30 is realized by, for example, a semiconductor memory device such as a RAM or flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 30 also has a multi-view video database 31 .

The multi-view video database 31 is a database that stores multi-view videos. FIG. 3 is a diagram showing an example of information stored in the multi-view video database 31. As shown in FIG. As shown in FIG. 3, the multi-view video database 31 stores items of information such as "video ID", "image group", "size information", "audio", and "sound source coordinates" in association with each other.

The “video ID” is an identifier for identifying each multi-view video, and the “image group” is data related to a group of images to be provided as a multi-view video. This data is a collection of moving images.

"Size information" indicates information about the size of the subject in the image group, for example, information about the actual size of the virtual object Ov. “Audio” indicates audio data to be played along with the multi-view video, and “sound source coordinates” is information regarding the coordinates of the sound source of the audio data in the multi-view video.

When an artist's singing video is provided as the AR content 100, the voice is the sound source of the artist singing, and the sound source coordinates are the head (for example, mouth) of the artist.

Returning to the description of FIG. 2, the control unit 40 will be described. The control unit 40 is, for example, a controller, and is stored in a storage device inside the information processing apparatus 10 by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processing Unit), or the like. Various programs are implemented by executing the RAM as a work area.

As shown in FIG. 2, the controller 40 includes a selector 41 and a distributor 42 . The selection unit 41 selects the corresponding multi-view video from the multi-view video database 31 based on the multi-view video distribution request received from each display control device 50 .

For example, the distribution request includes information about the video ID, and the selection unit 41 selects a multi-view video that matches the video ID included in the distribution request from the multi-view video database 31 .

Further, for example, during distribution of multi-view video, the selection unit 41 receives the user's point-of-regard information for the AR content 100 from each display control device 50, and performs viewpoint switching processing for the multi-view video based on the point-of-regard information. .

For example, the point-of-regard information is information about the imaging range and imaging orientation of the captured image G in the real space, and is information about the real space displayed in the AR content 100 . That is, it is information about the gaze point of the user in the AR content 100 . For example, when the user changes the position of the display control device 50 (for example, an outward facing camera 61 described later), the gaze point changes, and information about the changed direction and amount of change becomes the gaze point information.

The selection unit 41 selects a viewpoint position for the multi-view video based on the point-of-regard information, and selects a multi-view image corresponding to the selected viewpoint position from the multi-view video database 31, thereby performing viewpoint switching processing. In addition, the selection unit 41 performs a process of switching the viewpoint position at any time following changes in the point-of-regard information.

The distribution unit 42 distributes the multi-view video selected by the selection unit 41 to each display control device 50 . The multi-view video distributed by the distribution unit 42 is temporally continuous multi-view images. To deliver. In addition, the distribution unit 42 distributes the multi-view video together with information about audio and the like to the display control device 50 .

[2-2. Configuration example of display control device]
Next, a configuration example of the display control device 50 according to the embodiment will be described with reference to FIG. FIG. 4 is a block diagram showing a configuration example of the display control device 50. As shown in FIG. As shown in FIG. 4 , the display control device 50 includes a communication section 60 , an outward facing camera 61 , an inward facing camera 62 , a sensor group 63 , a display section 64 , an audio output section 65 , a storage section 70 and a control section 80 .

The communication unit 60 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 60 uses networks such as various wireless communication networks such as 4G (Generation), 5G, LTE (Long Term Evolution), Wifi (registered trademark) or wireless LAN (Local Area Network), or various wired communication networks. Information is sent and received to and from an external device via the network.

The outward camera 61 is a camera that captures an image of the outside of the display control device 50 (for example, the side opposite to the display surface of the display unit 64), and captures an image of the real space. For example, the captured image captured by the outward facing camera 61 is used as the background image of the virtual object Ov in the AR content 100 . The inward facing camera 61 is a camera that captures an image of the inside of the display control unit 50 (for example, the display surface side of the display unit 64), and captures an image of the user viewing the AR content 100. FIG.

The sensor group 63 is composed of various sensors for detecting, for example, the posture of the display control device 50, the three-dimensional shape of an object present in the surroundings, and the distance to the object. For example, the sensor group 63 includes a gyro sensor, lidar, and the like.

The display unit 64 is, for example, a touch panel display, displays various images such as the AR content 100, and receives various user operations. The audio output unit 65 is, for example, a surround speaker or a stereo speaker, and outputs various sounds that flow together with the AR content 100 .

The storage unit 70 is realized by, for example, a semiconductor memory device such as a RAM or flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 70 also has a map information storage unit 71 and a size information storage unit 72 .

The map information storage unit 71 stores map information of the real space. For example, the map information is an environment map indicating the surrounding environment of the display control unit 50, and is created based on the processing result by the self-position estimation unit 81, which will be described later.

Next, the controller 80 will be described. The control unit 80 is, for example, a controller, and is stored in a storage device inside the information processing apparatus 10 by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processing Unit), or the like. Various programs are implemented by executing the RAM as a work area.

As shown in FIG. 4 , the control unit 80 has a self-position estimation unit 81 , a display surface setting unit 82 , a display control unit 83 , an adjustment unit 84 and an audio control unit 85 . Based on various information input from the outward camera 61 and the sensor group 63, the self-position estimation unit 81 creates an environment map of the real space using a technique such as Slam (Simultaneous Localization and Mapping), and , the coordinates (for example, world coordinates) and orientation of the display control device 50 are estimated.

In addition, the self-position estimation unit 81 stores information on the created environment map as map information in the map information storage unit 71, and also stores information on the estimated position and orientation, that is, the above gaze point information, on the display surface setting unit 82. pass to Further, while the AR content 100 is being displayed, the point-of-regard information is also transmitted to the information processing device 10 .

The display surface setting unit 82 sets a virtual display surface (hereinafter simply display surface) for displaying the virtual object Ov in the real space based on the environment map created by the self-position estimation unit 81 .

For example, the display surface setting unit 82 searches for a display surface on which the virtual object Ov can be displayed from the environment map. Here, the display surface that can be displayed corresponds to, for example, the bottom surface of the display space that can contain the virtual object Ov displayed as the AR content 100 .

The display space here is a three-dimensional space that can include the virtual object Ov in the real space. That is, when displaying the virtual object Ov, a three-dimensional space that can be sufficiently displayed becomes the display space, and the bottom surface thereof becomes the display surface of the virtual object Ov.

As a result, for example, in the AR content 100, it is possible to avoid in advance an erroneous display in which the virtual object Ov is displayed through a wall or ceiling in the real space. Note that, for example, the display surface setting unit 82 may reduce the display size of the virtual object Ov in accordance with the real space so that the virtual object Ov fits in the real space. In other words, the display size of the virtual object Ov may be set according to the real space.

After setting the display space and the display surface, the display surface setting unit 82 writes various coordinate information regarding the display space and the display surface to the map information storage unit 71 . For example, the display surface is the floor of the real space, but it may be a part of the user's body (for example, the palm of the hand) depending on the type of the virtual object Ov. Alternatively, the display space itself may be made to float above the actual floor. Further, the display surface setting unit 82 may independently set the display space and the display surface for each virtual object Ov to be displayed. Further, for example, content in which the virtual object Ov itself moves within the AR content 100 is conceivable, so the display surface setting unit 82 sets the display space and the display surface so as to include the movement range of the virtual object Ov. may

After that, the display surface setting unit 82 updates the display size of the virtual object Ov according to the imaging position of the captured image G. FIG. For example, the display surface setting unit 82 recognizes the imaging position of the captured image G (for example, the position of the outward facing camera 61) based on the point-of-regard information received from the self-position estimating unit 81. FIG. Then, the display surface setting unit 82 sets the display size of the virtual object Ov based on the positional relationship between the imaging position and the display surface. Specifically, the display surface setting unit 82 increases the display size of the virtual object Ov as the imaging position and the display surface become closer, and makes the virtual object Ov smaller as the imaging position and the display surface become smaller. set the display size of

In this case, assuming that the virtual object Ov actually exists on the display surface, the display size of the virtual object Ov is updated so as to have the actual size shown in the captured image G.

The display control unit 83 superimposes the virtual object Ov on the captured image G to generate the AR content 100 and passes the generated AR content 100 to the adjustment unit 84 . For example, the display control unit 83 generates the AR content 100 by superimposing the virtual object Ov on the captured image G so that it is displayed on the display surface set by the display surface setting unit 82 .

The adjustment unit 84 adjusts the color of the captured image or the virtual object Ov in the AR content 100 according to the viewing mode of the AR content 100 displayed by the display control unit 83 .

Here, a specific example of processing by the adjustment unit 84 will be described with reference to FIGS. 5 and 6. FIG. 5 and 6 are diagrams showing an example of processing by the adjustment unit 84 according to the embodiment. The example shown in FIG. 5 shows a case where a virtual object Ov is displayed in the irradiation range of a lighting fixture L1 installed in real space.

In this case, for example, the adjustment unit 84 adjusts the color of the virtual object Ov as if the light from the lighting device L1 were illuminating the virtual object Ov. In this case, for example, the adjustment unit 84 detects the light of the lighting device L1 through various image analyzes of the captured image G, and adjusts the color of the virtual object Ov according to the detected light of the lighting device L1.

At this time, the adjustment unit 84 may detect the intensity of light and adjust the color of the virtual object Ov according to the intensity of the detected light. For example, the adjustment unit 84 may increase the color adjustment amount when the light intensity is high, and may decrease the color adjustment amount when the light intensity is high.

As a result, it is possible to reduce the color difference that occurs between the captured image G and the virtual object Ov. Also, in this case, it is possible to provide the AR content 100 in a display mode as if the virtual object Ov actually exists in the real space.

Although the case where the color of the virtual object Ov is adjusted based on the light in the real space has been described here, the color of the captured image G may be adjusted according to the light in the multi-viewpoint image, for example. Also, the colors of both the captured image G and the virtual object Ov may be adjusted according to the light in the real space or the multi-viewpoint image.

In the example of FIG. 6, another virtual object Ov2 is displayed behind the virtual object Ov1, and the viewpoint position for the virtual object Ov changes from the viewpoint position V1 to the viewpoint position V2. That is, the imaging position of the captured image G moves from the viewpoint position V1 to the viewpoint position V2, and the virtual object Ov2 is displayed at the viewpoint position V2 without being blocked by the virtual object Ov1.

In this case, for example, the virtual object Ov2 seen from the viewpoint position V1 may be partly shadowed by the virtual object Ov1. On the other hand, in such a situation, when viewing the virtual object Ov2 from the viewpoint position V2, it is difficult for the user to gaze at the virtual object Ov2 due to the shadow of the virtual object Ov1, even though the user wants to gaze at the virtual object Ov2.

Therefore, when displaying the virtual object Ov2 viewed from the viewpoint position V2, the adjustment unit 84 adjusts the color of the virtual object Ov2 and brightens the color of the virtual object Ov2, for example, thereby eliminating the shadow. I do. This allows the user to gaze at the virtual object Ov2 without being affected by shadows.

Also, the adjustment unit 84 may adjust the color of the captured image G or the virtual object Ov based on the positions of the light emitters virtually set for the AR content 100 .

For example, in this case, the user can set arbitrary lighting (illuminant) such as a spotlight or a mirror ball at an arbitrary position, and the adjustment unit 84 considers the light of the lighting to adjust the captured image G or the virtual object. The color of Ov may be adjusted.

As a result, for example, it is possible to provide AR content 100 with a lighting mode that the user likes, so that the variations of AR content 100 that can be provided can be increased.

Also, in this case, the adjustment unit 84 may move a virtual light emitter provided in the AR content 100 based on the user's operation to reflect the light of the light emitter in the AR content 100 .

7A and 7B are diagrams illustrating an example of an operation on a light emitter according to the embodiment; FIG. For example, as shown in FIG. 7 , the user can move the light emitter Sp within the AR content 100 by performing a slide operation on the display surface of the display control device 50 .

In the example shown in FIG. 7, the luminous body Sp is psyllium, and the adjustment unit 84 adjusts the captured image G and the virtual image as if the luminous body Sp were moving in the AR content 100 according to the position of the luminous body Sp. Controls the color of the object Ov.

As a result, for example, when the AR content 100 is a live video of music, it is possible for the user to feel as if he or she is actually participating in the live performance using psyllium.

Note that the adjustment unit 84 may move the position of the light emitter Sp in the AR content 100, for example, based on the target image of the user captured by the inward facing camera 62 (see FIG. 4). For example, an operation of actually shaking the psyllium by the user may be detected by image analysis of the target image, and the colors of the captured image G and the virtual object Ov may be adjusted according to the actual movement of the psyllium.

In this case, the colors of the captured image G and the virtual object Ov may be adjusted so that the light from the psyllium in the real space is projected onto the AR content 100 .

Alternatively, for example, a gyro sensor may be embedded in the psyllium in the real space, and based on the value of the gyro sensor, the colors of the captured image G and the virtual object Ov may be adjusted so as to reflect the light from the gyro sensor. .

Further, for example, the light of the psyllium is measured from an image of the psyllium in the real space captured by the inward-facing camera 62, and the colors of the captured image G and the virtual object Ov are adjusted so that the light is reflected in the AR content 100. may

Returning to the description of FIG. 4, the audio control unit 85 will be described. The audio control unit 85 controls audio output based on the positional relationship between the imaging position of the captured image G and the audio generation source set in the multi-viewpoint image. For example, the virtual object Ov is a person whose head is set as the sound source.

Here, for example, when the imaging position of the outward facing camera 61 changes, the position of the virtual object Ov in the AR content 100 changes, and the position of the sound source changes accordingly.

Therefore, the audio control unit 85 controls the output of the audio output unit 65 based on these positional relationships, thereby performing processing for matching the positions of the audio sources in the AR content 100 . Specifically, for example, when the virtual object Ov is displayed on the right side of the captured image G, the audio output unit 65 is controlled so that the sound can be heard from the user's right side, and the virtual object Ov is displayed on the front side of the captured image G. When displayed, the audio output unit 65 is controlled so that the audio can be heard from the near side.

In this way, the audio control unit 85 can provide highly realistic AR content 100 by controlling audio control according to changes in the position of the audio source. Note that the number of sound generation sources is not limited to one, and may be plural. In this case, the audio control unit 85 performs audio control based on the positional relationship for each audio source.

Further, for example, when there are a plurality of sound sources, the sound control unit 85 may perform various adjustments such as volume for each sound source, or integrate the sound sources. Also, the audio control unit 85 may receive an operation to change the position of the audio source, for example. In this case, for example, the sound source may be set at arbitrary coordinates different from the initially set sound source.

Also, the audio control unit 85 may delete, add, etc. the audio based on the user's operation, for example. That is, it is also possible to combine arbitrary sounds from a plurality of multi-viewpoint images and output them. In this case, for example, it is possible to provide AR content 100 or the like as if the voice sung by a different singer is sung by another singer's virtual object Ov.

[3. Processing procedure]
Next, processing procedures executed by the information processing device 10 and the display control device 50 according to the embodiment will be described with reference to FIGS. 8 and 9. FIG. First, a processing procedure executed by the information processing apparatus 10 will be described with reference to FIG. FIG. 8 is a flowchart showing a processing procedure executed by the information processing apparatus 10. As shown in FIG. Note that the processing procedure described below is repeatedly executed by the control unit 40 of the information processing apparatus 10 each time a distribution request is acquired.

As shown in FIG. 8, first, the information processing device 10 receives a distribution request (step S101). Subsequently, the information processing apparatus 10 distributes the multi-view video based on the received distribution request (step S102), and ends the process.

When distributing the multi-view video, the information processing apparatus 10 selects the multi-view image of the virtual object Ov whose viewpoint position is changed based on the point-of-regard information, and distributes the selected multi-view image.

Next, a processing procedure executed by the display control device 50 will be described with reference to FIG. FIG. 10 is a flow chart showing an example of a processing procedure executed by the display control device 50. As shown in FIG. The processing procedure described below is executed by the control unit 80 of the display control device 50 each time the AR content 100 is provided.

As shown in FIG. 9, first, the display control device 50 sets a display surface on which the virtual object Ov can be displayed in the real space based on, for example, a captured image of the surroundings (step S201).

Subsequently, the display control device 50 acquires the captured image G (step S202), and adjusts the color of the captured image G or the virtual object Ov (step S203). After that, the display control device 50 displays the AR content 100 using the color-adjusted captured image G or the virtual object Ov (step S204), and ends the process.

[4. Modification]
The display system S described above may be implemented in various different forms other than the above embodiment. Therefore, other embodiments of the display system S will be described below.

For example, the display control device 50 may be configured separately from the cameras (outward facing camera 61 and inward facing camera 62). The content provided by the display control device 50 is not limited to the AR content 100. For example, based on the captured image G, VR (Virtual Reality) content obtained by converting the real space into an animation or the like may be provided.

[5. effect〕
As described above, the display control device 50 according to the embodiment includes the display control unit 83 that displays a superimposed image in which a multi-viewpoint image is superimposed on a captured image of the surroundings, and an image that is displayed by the display control unit 83. and an adjustment unit 84 that adjusts the color of the captured image G or the multi-viewpoint image of the AR content 100 according to the viewing mode of the AR content 100 (an example of the superimposed image). Therefore, according to the display control device 50 according to the embodiment, it is possible to provide the AR content 100 suitable for the viewing mode.

Further, in the display control device 50 according to the embodiment, the display control unit 83 displays a superimposed image in which the viewpoint position of the multi-viewpoint image is controlled in accordance with the imaging position of the captured image G, and the adjustment unit 84 adjusts the viewpoint position. Adjust the colors accordingly. Therefore, according to the display control device 50 according to the embodiment, for example, the shadowed virtual object Ov can be displayed brightly, so that the AR content 100 that is highly convenient for the user can be provided.

In addition, in the display control device 50 according to the embodiment, the adjusting section 84 adjusts the color according to the ambient light control state. Therefore, according to the display control device 50 according to the embodiment, it is possible to provide the AR content 100 having a color that matches the dimming state of the real world, so that the sense of presence in the AR content 100 can be improved.

Further, in the display control device 50 according to the embodiment, the adjusting section 84 adjusts the color according to the intensity of light in the captured image. Therefore, according to the display control device 50 according to the embodiment, the stronger the light, the more the color can be adjusted. Therefore, the reality of the AR content 100 can be improved.

Also, in the display control device 50 according to the embodiment, the adjustment unit 84 adjusts the color based on the positions of the light emitters virtually set for the AR content 100 . Therefore, according to the display control device 50 according to the embodiment, since it is possible to arbitrarily set the light emitter in the AR content 100, it is possible to provide the AR content 100 in the user's favorite lighting environment.

In addition, in the display control device 50 according to the embodiment, the adjusting section 84 adjusts the color based on the user's operation on the light emitter. Therefore, according to the display control device 50 according to the embodiment, for example, when providing the AR content 100 such as live video, the user can participate in the live video by operating the light emitter.

Further, in the display control device 50 according to the embodiment, the adjustment unit 84 adjusts the color based on the position of the subject (for example, psyllium) included in the target image captured separately from the captured image G. Therefore, according to the display control device 50 according to the embodiment, the user can participate in the AR content 100 by operating the psyllium in the real world.

A display control method according to an embodiment is a display control method executed by a computer, and includes a display control step of displaying a superimposed image in which a multi-viewpoint image is superimposed on an imaged image of the surroundings; and an adjusting step of adjusting the colors of the captured image or the multi-viewpoint image in the superimposed image according to the viewing mode of the superimposed image displayed by the step.

In addition, the display control program according to the embodiment provides a display control procedure for displaying a superimposed image in which a multi-viewpoint image is superimposed on a captured image of the surroundings, and according to the viewing mode of the superimposed image displayed by the display control procedure. Then, the computer executes an adjustment procedure for adjusting the colors of the captured image or the multi-viewpoint image in the superimposed image.

[6. Hardware configuration]
Also, the display control device 50 according to the above-described embodiments is implemented by a computer 1000 configured as shown in FIG. 10, for example. FIG. 10 is a hardware configuration diagram showing an example of a computer that implements the functions of the display control device 50 according to the embodiment. Computer 1000 has CPU 1100 , RAM 1200 , ROM 1300 , HDD 1400 , communication interface (I/F) 1500 , input/output interface (I/F) 1600 and media interface (I/F) 1700 .

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, and the like. Communication interface 1500 receives data from other devices via network (communication network) N, sends the data to CPU 1100, and transmits data generated by CPU 1100 to other devices via network N. FIG.

Through an input/output interface 1600, the CPU 1100 connects an output device such as a display and a printer, and an input device such as a keyboard and a mouse (in FIG. 10, the output device and the input device are collectively referred to as "input/output device"). ). CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200 . CPU 1100 loads such a program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the display control device 50 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 120 by executing programs loaded on the RAM 1200 . The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired via the network N from another device.

As described above, some of the embodiments of the present application have been described in detail based on the drawings. It is possible to carry out the invention in other forms with modifications.

[7. others〕
Further, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or described as being performed manually. All or part of the processing can also be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing contents.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the self-position estimator 81 can be read as self-position estimator means or a self-position estimator circuit.

10 information processing device 20 communication unit 30 storage unit 31 multi-view video database 40 control unit 41 selection unit 42 distribution unit 50 display control unit 70 storage unit 71 map information storage unit 80 control unit 81 self-position estimation unit 82 display surface setting unit 83 Display control unit 84 Adjustment unit 85 Audio control unit 100 AR content G Captured image Ov Virtual object

Claims (9)

a display control unit that displays a superimposed image obtained by superimposing multi-viewpoint images obtained by photographing an existing subject from a plurality of viewpoints on a photographed image of the surroundings;
an adjustment unit that adjusts the color of the captured image or the multi-view image in the superimposed image according to the viewing mode of the superimposed image displayed by the display control unit;
The adjustment unit
A display control device , comprising: adjusting a color of an object displayed as the multi-viewpoint image according to a viewpoint position of the multi-viewpoint image, thereby correcting a shadow originally cast on the object. The display control unit
displaying the superimposed image in which the viewpoint position with respect to the multi-viewpoint image is controlled according to the imaging position of the captured image;
The adjustment unit
The display control device according to claim 1, wherein the color is adjusted according to the viewpoint position. The adjustment unit
3. The display control device according to claim 1, wherein the color is adjusted according to the dimming state of the surroundings. The adjustment unit
4. The display control device according to claim 1, wherein the color is adjusted according to the intensity of light in the captured image. The adjustment unit
5. The display control device according to any one of claims 1 to 4, wherein the color is adjusted based on a position of a light emitter that is virtually set with respect to the superimposed image. The adjustment unit
6. The display control device according to claim 5, wherein the color is adjusted based on an operation on the light emitter. The adjustment unit
The display control device according to any one of claims 1 to 6, wherein the captured image is adjusted based on the position of a subject included in a target image that is individually shot. A display control method executed by a computer,
a display control step of displaying a superimposed image in which multi-viewpoint images obtained by photographing an existing subject from a plurality of viewpoints are superimposed on a photographed image of the surroundings;
an adjusting step of adjusting the color of the captured image or the multi-view image in the superimposed image according to the viewing mode of the superimposed image displayed by the display control step;
The adjustment step includes
A display control method , comprising: adjusting a color of an object displayed as the multi-viewpoint image according to a viewpoint position of the multi-viewpoint image, thereby correcting a shadow originally cast on the object . Display control procedure for displaying a superimposed image obtained by superimposing multi-viewpoint images obtained by photographing an existing subject from a plurality of viewpoints on a photographed image of the surroundings When,
causing a computer to execute an adjustment procedure for adjusting the colors of the captured image or the multi-view image in the superimposed image according to the viewing mode of the superimposed image displayed by the display control procedure;
The adjustment procedure includes:
A display control program comprising: adjusting a color of an object displayed as the multi-viewpoint image according to a viewpoint position of the multi-viewpoint image, thereby correcting a shadow originally cast on the object.

Images