WO2023276156A1 - Spatial image display device, spatial image display method, and program - Google Patents

Abstract

A spatial image display device (1) of the present disclosure comprises: a plurality of display units (21) which display an image; a plurality of optical elements (22) which respectively reflect image light projected from the image displayed on the plurality of display units (21), and which are arranged so that the reflected image light reaches an observer's eye, thereby displaying a virtual image (VI) of the image, in the same direction from the observer and at a different distance; a determination unit (22) which determines a visual effect to be added to an object image containing an image of an object (OB), based on the position of the object (OB), which is a specific subject an image of which is included in the image; and a display control unit (34) which controls the display units (21) so that the image is displayed on the basis of the object image and the visual effect.

Description

SPATIAL IMAGE DISPLAY DEVICE, SPATIAL IMAGE DISPLAY METHOD, AND PROGRAM

The present disclosure relates to a spatial image display device, a spatial image display method, and a program.

A technique is known for displaying a virtual image in space by reflecting image light, which is light emitted from an image displayed on a display device, and allowing the reflected image light to reach the eyes of an observer.

It is also known to display a spatial image composed of a plurality of virtual images by displaying a plurality of virtual images at different distances in the same direction from the observer (Patent Document 1). This allows the observer to observe the spatial image with a higher stereoscopic effect. Also, by designing the distances between a plurality of virtual images to be short, the three-dimensional effect can be further enhanced.

Also, a stereoscopic image can be provided by displaying a plurality of images with different luminances at different distances in the same direction from the observer (Patent Document 2). As a result, the observer can see a stereoscopic image even at a position where no image exists, and as a result, the observer can observe the image with a high stereoscopic effect.

WO2017/038091 JP 2009-237310 A

However, in the technique described in Patent Document 1, the plurality of virtual images are virtual images formed on a two-dimensional virtual image plane. Therefore, in a configuration in which a plurality of virtual image planes are positioned at different distances in the same direction as viewed from the observer, if an object (for example, a ball used in a sports game) moves in a direction corresponding to the observation direction of the observer, , the virtual image of the object may be displayed on one virtual image plane at one time and displayed on another virtual image plane at another time. In such a case, the observer sees the virtual image of the object instantaneously moving from one virtual image plane to another, which seems unnatural compared to the movement of the object in real space. . Therefore, the observer may not be able to obtain a sufficient sense of realism.

In addition, when the technology described in Patent Document 2 is applied to an image showing a moving object to display a virtual image of the object on each of a plurality of virtual image planes, parallax between the plurality of virtual images may occur depending on the position of the observer. may occur and you may not be able to obtain a sufficient sense of presence.

An object of the present disclosure, which has been made in view of such circumstances, is to provide a spatial image display device, a spatial image display method, and a program capable of giving the observer a sufficient sense of realism when displaying virtual images of objects on a plurality of virtual image planes. to provide.

In order to solve the above problems, a spatial image display device according to the present disclosure includes a plurality of display units that display images, and image light emitted from the images displayed on the plurality of display units, respectively. a plurality of optical elements arranged such that a virtual image of the image is displayed at different distances in the same direction from the observer by the reflected image light reaching the eyes of the observer; a determination unit that determines a visual effect to be added to an object image including the image of the object based on the position of an object that is a specific subject whose image is included in the object; and determining the image based on the object image and the visual effect. a display controller for controlling the display to display.

Further, in order to solve the above problems, a spatial image display method according to the present disclosure includes a plurality of display units that display images, and image light emitted from the images displayed on the plurality of display units. and a plurality of optical elements arranged such that the reflected image light reaches an observer's eye, thereby displaying a virtual image of the image at different distances in the same direction from the observer. A spatial image display method for a spatial image display device comprising: determining a visual effect to be added to an object image including an image of the object based on the position of the object, which is a specific subject whose image is included in the image. and controlling the display to display the image based on the object image and the visual effect.

Also, in order to solve the above problems, a program according to the present disclosure causes a computer to function as the spatial image display device described above.

According to the spatial image display device, the spatial image display method, and the program according to the present disclosure, it is possible to give the observer a sufficient sense of realism when displaying virtual images of objects on a plurality of virtual image planes.

1 is a schematic diagram of a spatial image display device according to a first embodiment; FIG. 2 is a schematic diagram of a virtual image display unit shown in FIG. 1; FIG. 3 is a diagram for explaining a virtual image displayed by the virtual image display unit shown in FIG. 2; FIG. It is a figure which shows the virtual image seen from the observer. 2 is a diagram showing an example of visual effects stored in a visual effect storage section shown in FIG. 1; FIG. FIG. 4 is a diagram showing an example of the position of an object in real space; 6B is a diagram showing an example of a virtual image displayed based on the position of the object shown in FIG. 6A; FIG. FIG. 10 is a diagram showing another example of the position of an object in real space; 7B is a diagram showing an example of a virtual image displayed based on the position of the object shown in FIG. 7A; FIG. FIG. 10 is a diagram showing still another example of the position of an object in real space; 8B is a diagram showing an example of a virtual image displayed based on the position of the object shown in FIG. 8A; FIG. 2 is a flow chart showing an example of the operation in the spatial image display device shown in FIG. 1; FIG. 10 is a diagram showing a first modification of visual effects; 10B is a diagram showing an example of a virtual image displayed based on the visual effect shown in FIG. 10A; FIG. 10B is a diagram showing another example of a virtual image displayed based on the visual effect shown in FIG. 10A; FIG. FIG. 10 is a diagram showing a second modification of the visual effect; 11B is a diagram showing an example of a virtual image displayed based on the visual effect shown in FIG. 11A; FIG. 11B is a diagram showing another example of a virtual image displayed based on the visual effect shown in FIG. 11A. FIG. FIG. 11 is a diagram showing a third modification of visual effects; 12B is a diagram showing an example of a virtual image displayed based on the visual effect shown in FIG. 12A; FIG. 12B is a diagram showing another example of a virtual image displayed based on the visual effect shown in FIG. 12A; FIG. It is a schematic diagram of a spatial image display device according to a second embodiment. 14 is a sequence diagram showing an example of operations in the spatial image display device shown in FIG. 13; FIG. FIG. 11 is a schematic diagram of a spatial image display device according to a third embodiment; 16 is a sequence diagram showing an example of operations in the spatial image display device shown in FIG. 15; FIG. 2 is a hardware block diagram of a spatial image display device; FIG.

<<First Embodiment>>
The overall configuration of the first embodiment will be described with reference to FIG. FIG. 1 is a schematic diagram of a spatial image display device 1 according to the first embodiment.

As shown in FIG. 1, the spatial image display device 1 according to the first embodiment includes a virtual image display section 2 and a visual effect determination section 3.

<Configuration of virtual image display unit>
As shown in FIG. 2 , the virtual image display section 2 includes multiple display sections 21 and multiple optical elements 22 . The display unit 21 is configured by an organic EL (Electro Luminescence), a liquid crystal panel, or the like. The organic EL, liquid crystal panel, etc. may be attached to a device such as a tablet, VR goggles, or the like. The optical element 22 is composed of, for example, a half mirror that transmits part of the incident light and reflects the remaining part of the light. In the example shown in FIG. 2, the virtual image display unit 2 includes two plurality of display units 211 and 212 and two optical elements 221 and 222, but is not limited to this, and three or more display units 21 and 3 More than one optical element 22 may be provided.

The display unit 21 displays images. Specifically, the display unit 21 displays an image based on the object image under the control of the visual effect determination unit 3 . An object image is a part of an image generated by an imaging device such as a camera and includes an image of an object OB. The object OB is any object, but can be, for example, an object contained in an image that can be noticed by the observer. In an image generated by an imaging device capturing an image of a tennis match, the objects OB are, for example, a first player PL1, a second player PL2 who is an opponent of the first player PL1, and the first player PL2. The ball B is used by the second player PL1 and the second player PL2 during the game. Also, in this example, the imaging device is arranged so that the first player PL1 is positioned on the front side and the second player PL2 is positioned on the back side with respect to the imaging surface of the imaging device.

A video based on the object video includes a video in which the visual effect determined by the visual effect determination unit 3 is added to the object video. The image based on the object image may include an object image to which the visual effect determined by the visual effect determining unit 3 has not been added, that is, the object image itself. That is, the display unit 21 displays an image obtained by adding the visual effect determined by the visual effect determination unit 3 to the object image. Further, the display unit 21 may display an image to which no visual effect is added to the object image. The image obtained by adding a visual effect to the object image is an image obtained by processing the object image. The image is generated by processing the object image according to the type of visual effect, as will be described later in detail.

In the above-described example in which the imaging device generates a video image of a tennis match, as shown in FIG. Display the image IM1. The display unit 212 displays an object image IM2 including an image of the second player PL2, which is one of the objects OB. Further, the display unit 212 may display an object image IM3 including an image of a ball B, which is one of the objects OB, under the control of the visual effect determination unit 3. FIG. At this time, the display unit 211 does not have to display the object image IM3. Also, the display unit 211 may display an object image IM3. At this time, the display unit 212 does not have to display the object image IM3.

The optical elements 221 and 222 reflect image light emitted from the images displayed on the plurality of display units 211 and 212, respectively. The optical elements 221 and 222 are arranged such that the reflected image light reaches the observer's eye, thereby displaying a virtual image VI of the image at different distances in the same direction from the observer. In the examples shown in FIGS. 2 and 3, the distance from the observer to the virtual image plane VF1, which is one virtual image plane VF, and the distance from the observer to the virtual image plane VF2, which is the other virtual image plane VF, are different from each other. . Furthermore, the direction from the observer to the virtual image plane VF1 may be the same as the direction from the observer to the virtual image plane VF2. Located in The position of the virtual image plane VF1 can be calculated by a known method based on the positions of the display unit 211 and the optical element 221. is placed. For example, the virtual image plane VF1 is located on a plane symmetrical to the display surface of the display unit 211 with the optical element 221 interposed therebetween. The optical element 221 may be arranged so that Similarly, the position of the virtual image plane VF2 can be calculated by a known method based on the positions of the display unit 212 and the optical element 222. and an optical element 222 are arranged. For example, since the virtual image plane VF2 is positioned on a plane symmetrical to the display surface of the display unit 212 with the optical element 222 interposed therebetween, the display surface of the display unit 212 is symmetrical with respect to the desired position where the virtual image plane VF2 is displayed. The optical element 222 may be arranged so that

In the above-described example in which the imaging device images a tennis match, the optical elements 221 and 222 are arranged so that the virtual image planes VF1 and VF2 are positioned on the actual court. Specifically, the optical element 221 is arranged so that the virtual image plane VF1 is positioned closer to the observer than the actual net on the actual court. Also, the optical element 222 is arranged so that the virtual image plane VF2 is positioned farther from the observer than the actual net on the actual court. By combining the plurality of display units 211 and 212 and the optical elements 221 and 222 in this manner, a virtual image can be projected in a space in which the actual court, net, or the like is arranged. It is possible to enhance the three-dimensional effect possessed by.

As a result, as shown in FIG. 4, the observer observes the virtual image VI1 displayed on the virtual image plane VF1 on the near side, and observes the virtual image VI2 displayed on the virtual image plane VF2 on the far side. The virtual image VIb shown in FIG. 4 is displayed on either the virtual image plane VF1 or the virtual image plane VF2. The virtual image plane VF on which the virtual image VIb is displayed will be described later in detail. Therefore, the observer observes a spatial image composed of the virtual image VI1, the virtual image VI2, and the virtual image VIb. Such a virtual image is generally called a pepper's ghost or the like, and it looks to the observer as if the display portions 211 and 212 are on the virtual image planes VF1 and VF2, respectively.

<Configuration of visual effect determination unit>
As shown in FIG. 1, the visual effect determination unit 3 includes a communication unit 31, a visual effect storage unit 32, a determination unit 33, and a display control unit .

The communication unit 31 is configured by a communication interface. Standards such as Ethernet (registered trademark), FDDI (Fiber Distributed Data Interface), and Wi-Fi (registered trademark) may be used for the communication interface. The visual effect storage unit 32 is stored by memories such as HDD (Hard Disk Drive), SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory) and RAM (Random Access Memory). Configured. The determination unit 33 and the display control unit 34 constitute a control unit (controller). The control unit may be composed of dedicated hardware such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or may be composed of a processor, or may be composed of both. good too.

The communication unit 31 receives object video information indicating the object video and object position information indicating the position of the object OB from an external device. An object image is an image including an image of an object OB, which is a specific subject. The external device may be a video processing device that generates object video information and object position information, or may be any device that acquires object video information and object position information generated by a video processing device. .

As an example, the position of the object OB may be indicated by the distance in a predetermined direction from the reference plane, which is a plane in real space, corresponding to the virtual image plane VF on which the virtual image is displayed, to the object OB. The predetermined direction is the normal direction of the imaging surface of the camera that images the object. Note that the positional relationship of the specific subject viewed from the camera in a predetermined direction is substantially the same as the positional relationship of the virtual image of the specific subject viewed from the observer. Further, in the above-described example in which the imaging device captures an image of a tennis match, the predetermined direction is a predetermined position (for example, the center position) of the range in which the first player PL1 located on the near side from the imaging device exists. This is the direction in which the player is heading, and is the direction from the imaging device toward a predetermined position (for example, the center position) of the range where the second player PL2 exists.

The visual effect storage unit 32 associates and stores the position of the object OB in the real space with the visual effect. The visual effect is a visual effect given to the object image when the virtual image display unit 2 displays the virtual image VI of the object image indicated by the object image information. In the example shown in FIG. 5, the position is indicated by the distance from the reference plane to the object OB in real space, and the visual effect storage unit 32 stores the distance and the visual effect in association with each other.

The determination unit 33 determines a visual effect to be added to the object image including the image of the object OB, based on the position of the object OB, which is a specific subject whose image is included in the image. The position of the object OB may be the position of the object OB in the real space or the position of the image of the object OB in the video space. In the first embodiment, based on the object position information received by the communication unit 31 corresponding to the object image information received by the communication unit 31, the determination unit 33 adds to the object image indicated by the image information. Determine visual effects.

In the example shown in FIG. 5, the determination unit 33 determines the visual effect based on the distance from the reference plane to the object OB in real space. Specifically, the determination unit 33 extracts the visual effect stored in the visual effect storage unit 32 corresponding to the distance. Then, the determination unit 33 determines the visual effect to be added to the object video as the extracted visual effect.

In addition, in a configuration in which the position of the object OB is indicated by the distance from the reference plane to the object OB, the determination unit 33 causes the display unit 21 to display the object image from among the plurality of display units 21 based on the distance. may be determined. As an example, the determination unit 33 causes the display unit 21 that displays the object OB to display the virtual image VI on the virtual image plane VF corresponding to the reference plane with the shortest distance to the object OB among the plurality of reference planes. and decide.

The display control unit 34 controls the plurality of display units 21 to display images based on the object images. The virtual image display unit 2 controlled by the display control unit 34 may be variable, and may be, for example, a tablet, and may be replaced with VR goggles later. That is, the display control unit 34 may be compatible with multiple devices.

Specifically, the display control unit 34 controls the display unit 21 to display an image obtained by adding a visual effect to the image based on the object image. In this embodiment, the display control unit 34 causes the display unit to display an image obtained by adding the visual effect determined by the determination unit 33 to the object image of the object OB indicated by the object image information received by the communication unit 31. 21. At this time, in the configuration in which the determining unit 33 displays the object image from among the plurality of display units 21 based on the distance, the display control unit 34 determines whether the display unit 21 determined by the determining unit 33 is the object image. Control to display images with visual effects. In the example shown in FIG. 3, the display control unit 34 controls the display unit 21 determined by the determination unit 33 to display an image obtained by adding a visual effect to the object image IM3 including the image of the ball B that is the object OB. do.

Also, the display control unit 34 may control the predetermined display unit 21 to display the object video of the predetermined object OB indicated by the object video information. In the example shown in FIG. 3, the display control unit 34 controls the display unit 211 to display the object image IM1 including the image of the first player PL1, which is the predetermined object OB. The display control unit 34 also controls the display unit 212 to display the object image IM2 including the image of the second player PL2, which is the predetermined object OB.

Here, an example of the processing of the determination unit 33 and the display control unit 34 in the example shown in FIG. 5 will be described in detail.

In the example shown in FIG. 6A, the object video includes images of the first player PL1, the second player PL2, and the ball B, respectively. Also, the distance between the first reference plane S1 and the second reference plane S2 is 9 in the real space. In real space, a ball B, which is one of the objects, is positioned between the first reference plane S1 and the second reference plane S2, and the distance between the first reference plane S1 and the ball B is is 8.2.

As described above, the determination unit 33 determines the display unit 21 that displays the object OB as the display unit 21 that displays the virtual image VI on the virtual image plane VF corresponding to the reference plane with the shortest distance to the object OB. In this example, the distance from the first reference plane S1 to the ball B is 8.2, and the distance from the second reference plane S2 to the ball B is 0.8. Therefore, the determining unit 33 causes the display unit 21 that displays the virtual image VIb of the ball B to display the virtual image VI2 on the virtual image plane VF2 (see FIG. 3) corresponding to the second reference plane S2 with the shortest distance to the ball B. The display unit 212 to be displayed is determined.

Then, the determination unit 33 selects the visual effect “none” stored in the visual effect storage unit 32 as the object, corresponding to the distance 0.8 from the second reference plane S2 to the ball B, which is the object OB. Determine the visual effect to be added to OB.

The display control unit 34 controls the display unit 212 to display the object image IM3 including the image of the ball B, which is the object OB, without visual effects. Furthermore, the display control unit 34 controls the display unit 211 to display the object image IM1 including the image of the first player PL1, which is the predetermined object OB. The display control unit 34 also controls the display unit 212 to display the object image IM2 including the image of the second player PL2, which is the predetermined object OB. As a result, as shown in FIG. 6B, the observer observes the virtual image VI1 of the first player PL1 on the near side, and observes the virtual image VI2 of the second player PL2 and the virtual image VIb of the ball B on the far side. Therefore, the observer observes the virtual image VI1 of the first player PL1, the virtual image VI2 of the second player PL2, and the virtual image VIb of the ball B in a positional relationship similar to that of the respective objects OB in real space. be able to.

In the example shown in FIG. 7A, the object video includes images of the first player PL1, the second player PL2, and the ball B, similar to the example shown in FIG. 6A. Also, the distance between the first reference plane S1 and the second reference plane S2 is 9 in the real space. Further, in the real space, the ball B, which is one of the objects, is positioned between the first reference plane S1 and the second reference plane S2, and unlike the example shown in FIG. and ball B is 7.

In this example, the distance from the first reference plane S1 to the ball B is 7, and the distance from the second reference plane S2 to the ball B is 2. Therefore, the determination unit 33 causes the display unit 212 that displays the virtual image VI on the virtual image plane VF2 corresponding to the second reference plane S2, which is the shortest distance to the ball B, to display the object image including the image of the ball B. The display unit 21 is determined.

Then, the determining unit 33 sets the visual effect “enlargement/reduction” stored in the visual effect storage unit 32 to the object OB, corresponding to the distance 2 from the second reference plane S2 to the ball B, which is the object OB. Determine the visual effect to be added to OB. As an example, the determination unit 33 reduces the object image including the image of the ball B as the distance between the ball B and the second reference plane S2 becomes shorter, and reduces the object image including the image of the ball B as the distance increases. Decide to add visual effects such as magnifying. Note that the determination unit 33 determines the display unit 211 that displays the virtual image on the first reference plane S1 as the display unit 21 that displays the object image including the image of the ball B, and determines the display unit 21 that displays the object image including the image of the ball B. is within a predetermined range, the object image is enlarged as the distance between the ball B and the first reference surface S1 becomes shorter, and the object image is reduced as the distance becomes longer. may decide to do so.

Along with this, the display control unit 34 controls the display unit 212 to display an image to which a visual effect of enlarging or reducing the image of the ball B is added. The visual effect of enlarging or reducing is, for example, such that the image of ball B is enlarged as the ball B approaches the first player PL1, and the image of the ball B is reduced as the ball B moves away from the first player PL1. It can be a visual effect. Further, the display control unit 34 controls the display unit 211 to display the object image IM1 including the image of the first player PL1 that is the predetermined object OB, and the image of the second player PL2 that is the predetermined object OB. The display unit 212 is controlled to display the object image IM2 including As a result, the observer observes the virtual image VI1 of the first player PL1 on the near side, and the virtual image VI2 of the second player PL2 and the virtual image VIb of the ball B on the far side, as shown in FIG. 7B. Therefore, the observer can observe the virtual images of the first player PL1, the second player PL2, and the ball B in a positional relationship similar to their respective positional relationships in real space. Furthermore, since the virtual image display unit 2 reduces or enlarges the image of the ball B according to the change in distance, the observer can perceive the movement of the ball in the real space with a more realistic feeling.

In the example shown in FIG. 8A, the object video includes images of the first player PL1, the second player PL2, and the ball B, similar to the examples shown in FIGS. 6A and 7A. Also, the distance between the first reference plane S1 and the second reference plane S2 is 9 in the real space. Further, in the real space, the ball B, which is one of the objects, is positioned between the first reference plane S1 and the second reference plane S2, and unlike the examples shown in FIG. 6A and FIG. The distance between the reference plane S1 and the ball B is 5.

In this example, the distance from the first reference plane S1 to the ball B is 5, and the distance from the second reference plane S2 to the ball B is 4. For this reason, the determining unit 33 causes the display unit 21 that displays the object image including the image of the ball B to be displayed on the virtual image plane VF2 corresponding to the second reference plane S2 having the shortest distance to the ball B, which is the object OB. is determined to be the display unit 212 that displays .

Then, the determination unit 33 adds the visual effect “flash light” stored in the visual effect storage unit 32 to the object OB corresponding to the distance 2 from the second reference plane S2 to the ball B. Decide on the effect.

Along with this, the display control unit 34 displays the object image including the image of the ball B by adding a visual effect (shaded portion in FIG. 8B) that makes the image of the ball B look as bright as flash light. It controls the display unit 212 . Further, the display control unit 34 controls the display unit 211 to display the object image IM1 including the image of the first player PL1 that is the predetermined object OB, and the image of the second player PL2 that is the predetermined object OB. The display unit 212 is controlled to display the object image IM2 including As a result, the observer observes the virtual image VI1 of the first player PL1 on the near side, and the virtual image VI2 of the second player PL2 and the virtual image VIb of the ball B on the far side, as shown in FIG. 8B. Therefore, the observer can observe the virtual images of the first player PL1, the second player PL2, and the ball B in a positional relationship similar to their respective positional relationships in real space. Furthermore, the observer can more strongly recognize that the ball B is approaching by observing the image of the ball B to which the visual effect of the flash light is added.

<Operation of Spatial Image Display>
Here, the operation of the spatial image display device 1 according to the first embodiment will be described with reference to FIG. FIG. 9 is a flow chart showing an example of the operation of the spatial image display device 1 according to the first embodiment. The operation of the spatial image display device 1 described with reference to FIG. 9 corresponds to an example of the spatial image display method of the spatial image display device 1 according to the first embodiment.

In step S11, the communication unit 31 receives object video information and object position information from an external device via the communication network.

In step S12, the determining unit 33 determines a visual effect to be added to the object image, which is the image of the object included in the object image information, based on the position of the object corresponding to the object, which is the specific subject whose image is included in the image. decide.

In step S13, the display control unit 34 determines, from among the plurality of display units 21, the display unit 21 on which to display the object image including the image of the object OB.

In step S14, the display control unit 34 controls the display unit 21 to display the image based on the object image and the visual effect. Specifically, the display control unit 34 causes the display unit determined in step S13 to display an image obtained by adding the visual effect determined by the determination unit 33 to the object image of the object OB indicated by the object image information. 21. Furthermore, the display control unit 34 may control the predetermined display unit 21 to display the object image of the predetermined object OB indicated by the object image information.

In step S15, the display unit 21 displays the object image under the control of the display control unit 34. Specifically, the display unit 21 displays an image obtained by adding the visual effect determined by the determination unit 33 to the object image of the object OB indicated by the object image information. At this time, the display unit 21 may display an object image of a predetermined object.

As described above, according to the first embodiment, the spatial image display apparatus 1 adds an image of an object OB to an object image including an image of the object OB based on the position of the object, which is a specific subject whose image is included in the image. A visual effect is determined, and the display unit 21 is controlled to display an image based on the object image and the visual effect. Therefore, the spatial image display device 1 can display virtual images of objects on a plurality of virtual image planes VF. Therefore, the spatial image display device 1 can provide a sufficient sense of realism to an observer observing a spatial image formed by virtual images on a plurality of virtual image planes VF.

Further, according to the first embodiment, the spatial image display device 1 receives object video information indicating an object video and object position information indicating the position of the object OB, and based on the position indicated by the object position information, A visual effect to be added to the object video indicated by the object video information is determined. Therefore, the spatial image display device 1 can display a virtual image obtained by adding a visual effect to an object image included in the image, at a remote location where the object OB is imaged.

Also, according to the first embodiment, the spatial image display device 1 determines a visual effect such as changing the brightness of each object image based on the distance. Therefore, the spatial image display device 1 can display the virtual image VI of the moving object OB on a plurality of virtual image planes VF so that the observer can intuitively recognize the position of the object OB. It is possible to give a more realistic feeling to the observer who observes the stereoscopic image formed by.

Further, according to the first embodiment, the spatial image display device 1 determines, from among the plurality of display units 21, the display unit 21 on which the image based on the object image is to be displayed, based on the distance. The display unit 21 determined by is controlled to display an image obtained by adding a visual effect to the object image. Therefore, the spatial image display device 1 can display the virtual image VI of the moving object OB on a plurality of virtual image planes VF so that the observer can intuitively recognize the position of the object OB. It is possible to give a more realistic feeling to the observer who observes the stereoscopic image formed by.

Although the spatial image display device 1 includes the virtual image display unit 2 in the first embodiment described above, this is not the only option. For example, the spatial image display device 1 may not include the virtual image display unit 2 and may control an external virtual image display device to display the object image. In such a configuration, the spatial image display device 1 does not execute step S15 of the flowchart described above.

(First modification)
Also, in the above-described embodiment, the position of the object OB is indicated by the distance from the reference plane to the object OB in real space, but the present invention is not limited to this. For example, the position of object OB may be indicated by the height from a predetermined horizontal plane (for example, ground plane) to object OB in real space.

In such a configuration, the visual effect storage unit 32 stores heights and visual effects in association with each other, as shown in FIG. 10A. A determination unit 33 determines a visual effect based on the height. Furthermore, the display control unit 34 controls the display unit 21 to display an image obtained by adding the visual effect determined by the determination unit 33 to the object image.

In this example, when the height is 1.5, the determining unit 33 determines that the visual effect is "none" (see FIG. 10A). Then, the display control unit 34 controls the display unit 21 to display the object image without visual effects. This allows the observer to observe a spatial image formed by virtual images VI1, VI2, and VIb as shown in FIG. 10B.

Further, when the height is 2.5, the determination unit 33 determines the visual effect “50% transparency” (see FIG. 10A), and the determination unit 33 changes the transparency of the object image to 50%. The display unit 21 is controlled to display as As a result, the observer can observe a spatial image composed of the virtual images VI1 and VI2 as shown in FIG. 10C and the virtual image VIb of the object image with a visual effect of 50% transparency.

Also, when the height is 3 or more, the determination unit 33 determines the visual effect as “80% transparency” (see FIG. 10A), and changes the transparency of the object image to 80%. The display unit 21 is controlled to display

(Second modification)
In the above-described embodiment, the determining unit 33 determines the visual effect based on the position of the object OB. A visual effect may be determined. The change rate of the position can be, for example, the change in the position of the image of the object OB in a few unit frames forming the video.

In such a configuration, the visual effect storage unit 32 stores the rate of change in position and the visual effect in association with each other, as shown in FIG. 11A. The display control unit 34 also controls the display unit 21 to display an image obtained by adding the visual effect determined by the determination unit 33 to the object image.

In this example, when the position change rate is less than the threshold, the determination unit 33 determines the visual effect to be "none" (see FIG. 11A). Then, the display control unit 34 controls the display unit 21 to display the object image without visual effects. This allows the observer to observe a spatial image formed by virtual images VI1, VI2, and VIb as shown in FIG. 11B.

Also, when the rate of change in position is equal to or greater than the threshold, the determining unit 33 determines the visual effect to be "overlapping multiple frames" (see FIG. 11A). Then, the display control unit 34 controls the display unit 21 to display an image with a visual effect of superimposing and displaying a plurality of frames of object images. As a result, the observer can observe a spatial image composed of the virtual images VI1 and VI2 as shown in FIG. 11C and the virtual image VIb of the image displayed by overlapping the object images of a plurality of frames.

Also, when the change rate of the position cannot be determined, the determination unit 33 determines the visual effect as "50% transparency". Then, the display control unit 34 controls the display unit 21 to change the transparency of the object image to 50% and display it.

(Third modification)
Further, the determination unit 33 may determine the visual effect to be added to the object image based on whether the position of the object OB is within a predetermined range.

For example, as shown in FIG. 12A, the visual effect storage unit 32 stores visual effects in association with whether the position of the object OB is within a predetermined range. In the example shown in FIG. 12A, inside and on the line of a court used in a tennis match are within a predetermined range (IN), and outside the line is outside a predetermined range (OUT). The display control unit 34 also controls the display unit 21 to display an image obtained by adding the visual effect determined by the determination unit 33 to the object image.

In this example, when the position of the object OB in the real space is within the predetermined range, the determining unit 33 determines "no" visual effect. Then, the display control unit 34 controls the virtual image display unit 2 to display the object image without visual effects. This allows the observer to observe a spatial image formed by virtual images VI1, VI2, and VIb as shown in FIG. 12B.

Further, when the position of the object OB in the real space is outside the predetermined range, the determination unit 33 determines the visual effect "flash light". Then, the display control unit 34 controls the virtual image display unit 2 to display the object OB by adding a visual effect such that the image of the object OB looks bright like flash light. As a result, the observer can observe a spatial image composed of the virtual images VI1 and VI2 as shown in FIG. 12C and the virtual image VIb of the image added with the visual effect of making it look bright like flash light. .

Further, when it is unclear whether the position of the object OB in the real space is within the predetermined range, the determination unit 33 determines the visual effect as "50% transparency". Then, the display control unit 34 controls the virtual image display unit 2 so that the transparency of the object image is changed to 50% and displayed.

In the third modification, the determination unit 33 determines the visual effect based on whether the position of the object OB in the real space is within a predetermined range. A visual effect may be determined based on whether the image of is within a predetermined range.

(Fourth modification)
Further, in the above-described embodiment, the determination unit 33 determines the display unit 21 to display the object image based on the position of the object OB, but the present invention is not limited to this. For example, the determination unit 33 may determine to display the object image on each of the multiple display units 21 . In such a configuration, the position is indicated by the distance in a predetermined direction from the reference plane, which is a plane in real space, to the object, corresponding to the virtual image plane on which the image is displayed. , determine a visual effect that changes the brightness of each object image displayed on the plurality of display units 21 . Specifically, the determination unit 33 selects the object image to be displayed on each display unit 21 according to the position of the object OB so that the observer can observe the spatial image with a higher stereoscopic effect. The luminance is changed for each display section 21 .

(Fifth Modification)
Moreover, in the above-described embodiment, the determination unit 33 may further determine irradiation light to be applied to a member that defines the space in which the virtual image is displayed. Members that define the space in which the virtual image is displayed can be, for example, floors, walls, and pillars. The irradiation light is visible light. For example, the determination unit 33 may determine the color, intensity, etc. of the visible light, or may determine the projection image formed by the visible light according to the irradiation position. . In such a configuration, the display control unit 34 controls the irradiation device to emit the irradiation light determined by the determination unit 33 . The illumination device may be a lighting device, a projection device, or the like.

For example, in the example of the video of the tennis match as described above, when the ball B is positioned outside the line on the court, the determination unit 33 determines whether the ball B is positioned outside the line on the court. It is decided to irradiate the real coat with white light having a higher intensity than before. Then, the display control unit 34 controls the irradiation device to irradiate the coat with white light. In such an example, the observer can determine whether the ball B is positioned within or on the line on the court, or whether the ball B is on the line on the court, for example, due to the fast movement speed of the actual ball B. Even when it is not possible to clearly perceive whether the ball B is positioned outside, it is possible to instantly recognize the range on the court where the ball B is positioned by illuminating it with white light. Therefore, the observer can grasp the contents of the game without missing the timing.

<<Second Embodiment>>
The overall configuration of the second embodiment will be described with reference to FIG. FIG. 13 is a schematic diagram of a spatial image display device 1-1 according to the second embodiment. In the second embodiment, functional units that are the same as those in the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.

As shown in FIG. 13, the spatial image display device 1-1 according to the second embodiment includes a virtual image display section 2, a visual effect determination section 3-1, and a video processing section 4-1.

<Configuration of video processing unit>
The video processing unit 4-1 includes an input unit 41 and an object extraction unit . The input unit 41 is configured by an input interface that receives input of information. The object extraction unit 42 constitutes a control unit.

The input unit 41 accepts input of image information indicating an image generated by the imaging device.

The object extraction unit 42 extracts the image of the object OB from the image information whose input is accepted by the input unit 41 . Any method may be used for the object extraction unit 42 to extract the object video.

<Configuration of visual effect determination unit>
As shown in FIG. 1, the visual effect determination unit 3-1 includes a communication unit 31-1, a visual effect storage unit 32, a determination unit 33-1, and a display control unit .

The communication unit 31-1 receives object position information. Specifically, it receives object position information from an external device via a communication network.

The determination unit 33-1 determines the visual effect to be added to the object video including the image of the object OB based on the position of the object OB. Specifically, the determination unit 33-1 determines the visual effect to be added to the object video extracted by the object extraction unit 42 based on the position of the object OB indicated by the object position information received by the communication unit 31-1. decide. A specific method for determining the visual effect by the determining unit 33-1 is the same as the specific method for determining the visual effect by the determining unit 33 in the first embodiment described above.

<Operation of Spatial Image Display>
Here, the operation of the spatial image display device 1-1 according to the second embodiment will be described with reference to FIG. FIG. 14 is a flow chart showing an example of the operation of the spatial image display device 1-1 according to the second embodiment. The operation of the spatial image display device 1-1 described with reference to FIG. 14 corresponds to an example of the spatial image display method of the spatial image display device 1-1 according to the second embodiment.

In step S21, the input unit 41 accepts input of video information indicating the video generated by the imaging device.

In step S22, the object extraction unit 42 extracts object video information indicating an object video including the image of the object OB from the video information.

In step S23, the communication unit 31-1 receives object position information from an external device via the communication network.

Subsequently, the spatial image display device 1-1 executes the processing from step S24 to step S27. The processing from step S25 to step S28 is the same as the processing from step S12 to step S15 in the first embodiment.

Note that in the second embodiment, the visual effect determination unit 3-1 and the video processing unit 4-1 may be configured separately. In such a configuration, the video processing unit 4-1 has a communication unit configured by a communication interface, and the communication unit determines the visual effect of the object video information indicating the object video extracted by the object extraction unit 42. It is transmitted to the communication section 31-1 of the section 3-1.

Also, the first to fifth modifications of the first embodiment described above may be applied to the second embodiment.

<<Third Embodiment>>
The overall configuration of the third embodiment will be described with reference to FIG. FIG. 15 is a schematic diagram of a spatial image display device 1-2 according to the third embodiment. The same reference numerals are given to the same functional units as in the second embodiment, and the description thereof is omitted.

As shown in FIG. 15, the spatial image display device 1-2 according to the second embodiment includes a virtual image display section 2, a visual effect determination section 3-2, and an image processing section 4-2.

<Configuration of video processing unit>
The video processing unit 4-2 includes an input unit 41, an object extraction unit 42, and an object position estimation unit 43.

The object position estimation unit 43 estimates the position of the object OB. Any method may be used by the object position estimation unit 43 to estimate the position of the object OB. For example, the object position estimation unit 43 estimates the distance from the reference plane in the real space to the object OB using deep learning, which indicates the position of the object OB based on the video information input by the input unit 41. good too.

<Configuration of visual effect determination unit>
As shown in FIG. 15, the visual effect determination unit 3-2 includes a visual effect storage unit 32, a determination unit 33-2, and a display control unit .

The determination unit 33-2 determines a visual effect to be added to the object image including the image of the object OB, based on the position of the object OB. Specifically, the determination unit 33 - 2 determines the visual effect to be added to the object image extracted by the object extraction unit 42 based on the position estimated by the object position estimation unit 43 . A specific method for determining the visual effect by the determining unit 33-2 is the same as the specific method for determining the visual effect by the determining unit 33 in the first embodiment described above.

<Operation of Spatial Image Display>
Here, the operation of the spatial image display device 1-2 according to the third embodiment will be described with reference to FIG. FIG. 16 is a sequence diagram showing an example of the operation of the spatial image display device 1-2 according to the third embodiment. The operation of the spatial image display device 1-2 described with reference to FIG. 16 corresponds to an example of the spatial image display method of the spatial image display device 1-2 according to the third embodiment.

In step S31, the input unit 41 accepts input of image information indicating an image generated by an imaging device such as a camera.

In step S32, the object extraction unit 42 extracts object video information indicating the video of the object OB from the video information.

In step S33, the object position estimation unit 43 estimates the object position indicating the position of the object OB.

Subsequently, the spatial image display device 1-2 executes the processing from step S34 to step S37. The processing from step S34 to step S37 is the same as the processing from step S12 to step S15 in the first embodiment.

Note that the first to fifth modifications of the first embodiment described above may be applied to the third embodiment. In the configuration to which the first modification is applied, the object position estimation unit 43 estimates the height from a predetermined horizontal plane to the object OB in real space, which indicates the position of the object OB. In addition, in the configuration to which the second modification is applied, the object position estimation unit 43 estimates the change rate of the position of the image of the object OB in the video. In the configuration to which the third modification is applied, the object estimation unit 44 estimates whether or not the position of the object OB in real space is within a predetermined range.

<Program>
The determination units 33 , 33 - 1 , 33 - 2 and the display control unit 34 described above can be realized by the computer 100 . Further, a program for functioning as the determination units 33, 33-1, 33-2 and the display control unit 34 may be provided. Also, the program may be stored in a storage medium or provided through a network. FIG. 17 is a block diagram showing a schematic configuration of the computer 100 functioning as the determination units 33, 33-1, 33-2, and the display control unit 34, respectively. Here, the computer 100 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. Program instructions may be program code, code segments, etc. for performing the required tasks.

As shown in FIG. 17, the computer 100 includes a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a storage 140, an input unit 150, a display unit 160, and a communication interface ( I/F) 170. Each component is communicatively connected to each other via a bus 180 . The processor 110 is specifically a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc. may be configured by a plurality of processors of

The processor 110 controls each configuration and executes various arithmetic processing. That is, processor 110 reads a program from ROM 120 or storage 140 and executes the program using RAM 130 as a work area. The processor 110 performs control of each configuration and various arithmetic processing according to programs stored in the ROM 120 or the storage 140 . In the above-described embodiment, the ROM 120 or storage 140 stores the program according to the present disclosure.

The program may be stored in a storage medium readable by the computer 100. A program can be installed in the computer 100 by using such a storage medium. Here, the storage medium storing the program may be a non-transitory storage medium. The non-temporary storage medium is not particularly limited, but may be, for example, a CD-ROM, a DVD-ROM, a USB (Universal Serial Bus) memory, or the like. Also, this program may be downloaded from an external device via a network.

The ROM 120 stores various programs and various data. RAM 130 temporarily stores programs or data as a work area. The storage 140 is configured by a HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including an operating system and various data.

The input unit 150 includes one or more input interfaces that receive user's input operations and acquire information based on the user's operations. For example, the input unit 150 is a pointing device, keyboard, mouse, etc., but is not limited to these.

The display unit 160 includes one or more output interfaces that output information. For example, the display unit 160 is a display that outputs information as video or a speaker that outputs information as audio, but is not limited to these. Note that the display unit 160 also functions as the input unit 150 when it is a touch panel type display.

A communication interface (I/F) 170 is an interface for communicating with an external device.

Regarding the above embodiments, the following additional remarks are disclosed.

(Appendix 1)
a plurality of display units for displaying images;
The image light emitted from the images displayed on the plurality of display units is reflected, and the reflected image light reaches the observer's eye, whereby the image is displayed at different distances in the same direction from the observer. a plurality of optical elements arranged to display a virtual image of an image;
a control unit;
The control unit
determining a visual effect to be added to an object image including an image of the object based on the position of an object that is a specific subject whose image is included in the image;
A spatial image display device for controlling the display unit to display the image based on the object image and the visual effect.
(Appendix 2)
a communication unit that receives object video information indicating the object video and object position information indicating the position of the object;
2. The spatial image display device according to claim 1, wherein the control unit determines the visual effect to be added to the object image indicated by the object image information based on the position indicated by the object position information.
(Appendix 3)
The control unit
estimating the position of the object;
3. The aerial image display device according to claim 1 or 2, wherein the visual effect is determined based on the position.
(Appendix 4)
The position is indicated by the distance from a reference plane, which is a plane in real space, to the object, corresponding to the virtual image plane on which the virtual image is displayed,
The control unit
determining, from among the plurality of display units, a display unit on which to display the object image based on the distance;
4. The spatial image display device according to any one of additional items 1 to 3, wherein the display unit determined by the determination unit controls to display the image added with the visual effect to the object image.
(Appendix 5)
The position is indicated by a height from a predetermined horizontal plane to the object in real space,
5. The spatial image display device according to any one of additional items 1 to 4, wherein the control unit determines the visual effect based on the height.
(Appendix 6)
6. The aerial image display device according to any one of additional items 1 to 5, wherein the control unit determines the visual effect based on the change rate of the position.
(Appendix 7)
7. The aerial image display device according to any one of additional items 1 to 6, wherein the control unit determines the visual effect based on whether the position is within a predetermined range.
(Appendix 8)
The position is indicated by a distance in a predetermined direction from a reference plane, which is a plane in real space, to the object, corresponding to the virtual image plane on which the virtual image is displayed;
4. The spatial image display device according to claim 3, wherein the control unit determines the visual effect of changing the brightness of each of the object images displayed on the plurality of display units, based on the distance.
(Appendix 9)
The control unit
Determining irradiation light irradiated to a member defining a space in which the virtual image is displayed;
9. The spatial image display device according to any one of additional items 1 to 8, wherein an irradiation device is controlled to irradiate the determined irradiation light.
(Appendix 10)
10. The spatial image display device according to any one of additional items 1 to 9, wherein the control unit is compatible with multiple devices.
(Appendix 11)
a plurality of display units that display images; and image light emitted from the images displayed on the plurality of display units are reflected, respectively, and the reflected image light reaches an observer's eye. A spatial image display method for a spatial image display device comprising a plurality of optical elements arranged such that virtual images of the video are displayed at different distances in the same direction from an observer,
determining a visual effect to be added to an object image including an image of the object based on the position of the object, which is a specific subject whose image is included in the image;
controlling the plurality of displays to display the image based on the object image and the visual effect;
a spatial image display method comprising:
(Appendix 12)
A non-temporary storage medium storing a program executable by a computer, the non-temporary storage medium storing the program causing the computer to function as the determining unit and the display control unit according to any one of appendices 1 to 10. storage medium.

All publications, patent applications and technical standards mentioned herein are expressly incorporated herein by reference to the same extent as if each individual publication, patent application and technical standard were specifically and individually indicated to be incorporated by reference. incorporated herein by reference.

Although the above-described embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be construed as limited by the above-described embodiments, and various modifications and changes are possible without departing from the scope of the claims. For example, it is possible to combine a plurality of configuration blocks described in the configuration diagrams of the embodiments into one, or divide one configuration block.

1, 1-1, 1-2 Spatial image display device 2 Virtual image display unit 3, 3-1, 3-2 Visual effect determination unit 4-1, 4-2 Video processing unit 21, 211, 212 Display unit 22, 221 , 222 optical elements 31, 31-1 communication unit 32 visual effect storage units 33, 33-1, 33-2 determination unit 34 display control unit 41 input unit 42 object extraction unit 43 object position estimation unit 100 computer 110 processor 120 ROM
130 RAM
140 storage 150 input unit 160 output unit 170 communication interface 180 bus

Claims (12)

1. a plurality of display units for displaying images;
   The image light emitted from the images displayed on the plurality of display units is reflected, and the reflected image light reaches the observer's eye, whereby the image is displayed at different distances in the same direction from the observer. a plurality of optical elements arranged to display a virtual image of an image;
   a determination unit that determines a visual effect to be added to an object image including the image of the object based on the position of the object, which is a specific subject whose image is included in the image;
   a display control unit that controls the display unit to display the image based on the object image and the visual effect;
   A spatial image display device comprising:
2. a communication unit that receives object video information indicating the object video and object position information indicating the position of the object;
   2. The spatial image display device according to claim 1, wherein said determining unit determines said visual effect to be added to said object image indicated by said object image information based on said position indicated by said object position information.
3. Further comprising an object position estimator that estimates the position of the object,
   3. The aerial image display device according to claim 1, wherein said determining section determines said visual effect based on said position estimated by said object position estimating section.
4. The position is indicated by the distance from a reference plane, which is a plane in real space corresponding to the virtual image plane on which the virtual image is displayed, to the object,
   The determining unit determines, from among the plurality of display units, a display unit on which to display an image based on the object image based on the distance,
   4. The aerial image according to any one of claims 1 to 3, wherein said display control unit controls said display unit determined by said determining unit to display an image obtained by adding said visual effect to said object image. display device.
5. The position is indicated by a height from a predetermined horizontal plane to the object in real space,
   The aerial image display device according to any one of claims 1 to 4, wherein the determination unit determines the visual effect based on the height.
6. The aerial image display device according to any one of claims 1 to 5, wherein the determination unit determines the visual effect based on the change rate of the position.
7. The spatial image display device according to any one of claims 1 to 6, wherein the determination unit determines the visual effect based on whether the position is within a predetermined range.
8. The position is indicated by a distance in a predetermined direction from a reference plane, which is a plane in real space corresponding to the virtual image plane on which the virtual image is displayed, to the object,
   4. The spatial image display device according to claim 3, wherein said determining unit determines said visual effects such as changing luminance of said object images displayed on said plurality of display units based on said distance.
9. The determining unit determines irradiation light to be applied to a member that defines a space in which the virtual image is displayed,
   The spatial image display device according to any one of claims 1 to 8, wherein the display control section controls an irradiation device to emit the irradiation light determined by the determination section.
10. The spatial image display device according to any one of claims 1 to 9, wherein the display control unit is compatible with multiple devices.
11. a plurality of display units that display images; and image light emitted from the images displayed on the plurality of display units are reflected, respectively, and the reflected image light reaches an observer's eye. A spatial image display method for a spatial image display device comprising a plurality of optical elements arranged such that virtual images of the video are displayed at different distances in the same direction from an observer,
    determining a visual effect to be added to an object image including an image of the object based on the position of the object, which is a specific subject whose image is included in the image;
    controlling the display to display the image based on the object image and the visual effect;
    a spatial image display method comprising:
12. A program for causing a computer to function as the determination unit and the display control unit according to any one of claims 1 to 10.

Images