JP2023032393A - Image display device, image display method, and image display system - Google Patents

Abstract

To display an image capable of giving a user both presence about appreciation of an exhibition, an event, etc. and reality about movement between appreciation areas such as exhibition or event performance areas.SOLUTION: An image display device comprises a display unit and a display control unit which controls the display unit. The display control unit causes the display unit to switch and display a real time image in appreciation areas imaged by a camera mounted on a travel object travelling in the appreciation areas and a travel path image based on a recorded image captured in the past on a travel path between appreciation areas.SELECTED DRAWING: Figure 1

Description

The present invention relates to technology for displaying images.

When actually visiting an exposition having a plurality of exhibition areas, a theme park having a plurality of event areas, etc., there is a problem that it is tiring to move between areas.

In order to solve the above problem, it is conceivable to simulate participation in an exposition or a theme park by viewing real-time video within an area at a remote location such as one's home or a satellite venue.

In a video display system in which a real-time video within an area can be viewed at a remote location, a configuration is conceivable in which real-time video within each area is instantaneously switched like a multi-dimensional live relay relay in television broadcasting.

JP-A-5-96062

In a video display system configured to instantaneously switch between real-time videos in each area, movement between areas becomes like instantaneous movement, and the user cannot be given a sense of reality regarding movement between areas.

Note that the virtual experience apparatus disclosed in Patent Document 1 cannot give the user a sense of realism related to an exhibition or event because all the images shown to the user are recorded images.

In view of the above problems, the present invention provides a sense of realism related to appreciation of exhibitions or events, and a viewing area such as an exhibition or event implementation area (an area similar to an area where visitors can enjoy viewing and experiencing exhibits and events). It is an object of the present invention to provide a technique for displaying an image that can give a user both a sense of reality regarding movement between viewing areas.

An exemplary video display device of the present invention includes a display section and a display control section that controls the display section. The display control unit moves based on a real-time image within the viewing area captured by a camera mounted on a moving object moving within the viewing area and a recorded image captured in the past on a moving route between the viewing areas. The route image is switched to be displayed on the display section.

According to the exemplary present invention, it is possible to give the user both a sense of realism related to viewing an exhibition or event, and a sense of reality related to movement between viewing areas.

1 is a block diagram showing the configuration of a video display system according to an embodiment; FIG. 4 is a flow chart showing an operation example of the video display system according to the embodiment; FIG. 4 is a diagram schematically showing an example of linking information between a video display device and a robot; FIG. 11 is a diagram showing an example of transition of images displayed by the display unit; A diagram schematically showing the positional relationship between the viewpoint of the moving route video and the entrance to the second exhibition area. Diagram showing the positional relationship between the entrance to the second exhibition area and the robot

Exemplary embodiments of the invention are described in detail below with reference to the drawings.

<Configuration of image display system>
FIG. 1 is a block diagram showing the configuration of a video display system 1 according to the embodiment. The video display system 1 is a system used for a user to virtually participate in an exposition having a plurality of exhibition areas, a theme park having a plurality of event areas, or the like.

Here, a case will be described in which the image display system 1 is used for a user to virtually participate in an exposition having a plurality of exhibition implementation areas. The exhibition implementation area may be set for each pavilion, or may be set for each section within the pavilion.

In order to simplify the explanation, it is assumed that the exposition has only the first exhibition area and the second exhibition area.

The image display system 1 includes an image display device 10 , robots 20</b>A and 20</b>B, and an image providing device 30 .

The video display device 10 includes a display control section 11 , a display section 12 , an input section 13 , a communication section 14 , a speaker 15 and a microphone 16 . The video display device 10 is installed at a user's home, a satellite site of an exposition, or the like. Although only one image display device 10 is shown in FIG. 1, the image display system 1 includes a plurality of image display devices 10 so that a plurality of users can use the image display system 1 at the same time.

The display control unit 11 controls the display unit 12 to display the image provided from the image providing device 30 on the display unit 12 . The display control unit 11 includes, for example, a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), input/output ports, and various circuits. The display control unit 11 performs voice control, communication control, input data processing, etc., in addition to display control.

As the display unit 12, for example, a thin display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel can be used. When at least part of the video provided by the video providing device 30 is a VR (Virtual Reality) video, the display unit 12 may be VR glasses, VR goggles, or the like, for example.

The input unit 13 inputs information related to remote control of the robots 20A and 20B, instruction information related to moving route images, and the like. As the input unit 13, for example, a keyboard, pointing device, touch panel, joystick, etc. can be used.

The communication unit 14 performs data communication with the image providing device 30 via a network (not shown).

The speaker 15 restores and outputs the sound collected by the microphone 25 of the robot 20A or 20B. The microphone 16 collects ambient sounds of the video display device 10 . The sound collected by the microphone 16 is restored and output by the speaker 26 of the robot 20A or 20B. As a result, for example, a user using the video display device 10 can interact with a person in the first exhibition area or the second exhibition area. Alternatively, the microphone 16 and the speaker 26 may be omitted so that the user using the video display device 10 can only listen to the sounds generated in the first exhibition area or the second exhibition area. Alternatively, the user using the video display device 10 may enjoy only the video by omitting the speaker 15, the microphone 16, the microphone 25, and the speaker 26. FIG.

The robot 20A is placed in the first exhibition implementation area. Although only one robot 20A is shown in FIG. 1, the image display system 1 uses the robot 20A so that each of a plurality of users can simultaneously view a real-time image of a desired location within the first exhibition implementation area. Have multiple.

Robot 20B is placed in the second exhibition implementation area. Although only one robot 20B is shown in FIG. 1, the image display system 1 uses the robot 20B so that each of a plurality of users can simultaneously view a real-time image of a desired location within the second exhibition area. Have multiple.

Since the robots 20A and 20B have the same configuration, only the robot 20A will be described. Note that, unlike the present embodiment, the robot 20A and the robot 20B may have slightly different configurations due to the addition or reduction of functions, enhancement of functions, or the like.

The robot 20</b>A includes a control section 21 , a drive section 22 , a camera 23 , a communication section 24 , a microphone 25 and a speaker 26 . The robot 20A is a self-propelled robot. And the robot 20A can be remotely controlled. The position of the robot 20A can be moved by remote control of the robot 20A, and as a result, it becomes easy for the user to virtually move to a desired location within the first exhibition implementation area.

The control unit 21 controls the entire robot 20A according to remote control.

The drive unit 22 drives the wheels and the like based on the control of the control unit 21, and changes the movement direction/speed of the robot 20A, the orientation of the robot 20A, and the like.

Camera 23 shoots an image. The camera 23 may be configured to be movable with respect to the body of the robot 20A, and the direction of the camera 23 may be changed by remote control even when the robot 20A is in a stopped state.

The communication unit 24 performs data communication with the image providing device 30 via a network (not shown).

The microphone 25 collects sounds around the robot 20A. The speaker 26 restores and outputs the sound collected by the microphone 16 of the video display device 10 . The robot 20A also collects past images, which will be described later, while being remotely controlled. The robot 20A may also collect other operating states when collecting past images, which will be described later. That is, the robot 20A transmits image data including position data, shooting method data, etc. as meta information to the image providing apparatus 30 as an image for generating a past image.

The image providing device 30 includes a control section 31 , a recording section 32 and a communication section 33 . Note that the image providing device 30 may be, for example, a server device installed at a single location, or a distributed server device having constituent elements distributed and installed at a plurality of locations.

The control unit 31 controls the image providing device 30 as a whole. The control unit 31 includes, for example, a computer having a CPU, ROM, RAM, HDD, input/output ports, and various circuits.

The recording unit 32 records recorded images that have been taken in the past along the moving route between the first exhibition area and the second exhibition area.

The recording unit 32 also stores map information of the venues of the exposition (excluding satellite venues).

<Operation of video display system>
Next, the operation of the image display system 1 will be described. FIG. 2 is a flowchart showing an operation example of the video display system 1. As shown in FIG.

For example, when the input unit 13 inputs information indicating the start of use of the system, the operation of the flowchart shown in FIG. 2 is started. The operation of the flowchart shown in FIG. 2 is terminated when the input unit 13 inputs information indicating termination of use of the system.

First, in step S<b>10 , the video display device 10 selects an exhibition implementation area to enter in a pseudo manner based on information input to the input unit 13 .

In the next step S<b>20 , the image providing device 30 associates the image display device 10 with the robot 20</b>A or 20</b>B and records the association information in the recording unit 32 . The robot 20A or 20B to be linked to the video display device 10 is assumed to be a robot that is not linked to another video display device 10 at the time of linking (empty robot).

FIG. 3 is a diagram schematically showing an example of linking information among a robot, a video display device, and a robot. The robot 20A whose ID is "111111" is associated with the video display device 10 whose ID is "KKKK". The robot 20A whose ID is "222222" is an idle robot. The robot 20B with an ID of "333333" is associated with the video display device 10 with an ID of "BBBB". Note that the placement area in the example shown in FIG. It may be a moving route. The coordinates indicating the current position of the robot include latitude information and longitude information, for example. The current position data of the robot can be used for various controls such as displaying the current position of the robot on a map.

When the linking in step S20 is completed, the real-time video captured by the camera 23 of the robot 20A or 20B linked to the video display device 10 is relayed by the router function of the video providing device 30, and the video display device 10 is displayed on the display unit 12 (step S30). Incidentally, if the robot 20A and the video display device 10 are associated with each other so that the data for connection processing with the communication partner such as the IP address of each other can be known, the transfer of the real-time video does not involve the video providing device 30. It is also possible for the robot 20A and the video display device 10 to directly perform this.

By operating the input unit 13 during execution of step S<b>30 , the user can remotely control the robot 20</b>A or 20</b>B to which the image display device 10 is linked.

In the next step S40, the image providing device 30 determines whether or not the user will pseudo-exit from the exhibition area that the user is pseudo-entering. For example, when the robot 20A or 20B linked to the image display device 10 moves to the exit of the exhibition area that the user is currently entering by remote control, the image providing device 30 allows the user to enter the exhibition area in a pseudo manner. You can judge that you will leave the exhibition area in the middle.

When it is determined in step S40 that the user does not pseudo-exit, the process of step S30 is continued. On the other hand, if it is determined in step S40 that the user will leave the venue in a pseudo manner, the process proceeds to step S50.

In step S50, the image providing device 30 releases the link between the image display device 10 and the robot 20A or 20B.

In the next step S<b>60 , the image display device 10 displays the moving route image provided by the image providing device 30 on the display section 12 .

Here, an outline of the moving route image will be described.

The video providing device 30 generates a moving route video based on the recorded video captured in the past along the moving route between the first exhibition area and the second exhibition area. In other words, the image providing device 30 edits the recorded images taken in the past along the movement route between the first exhibition area and the second exhibition area to generate the movement route image. The image providing device 30 may determine the contents of editing based on the instruction information input to the input unit 13 . The instruction information input to the input unit 13 includes, for example, the position in the moving route, the viewpoint of the image, the viewing direction of the image, and the like.

If the moving speed of the camera that shot the recorded video is not constant, the video providing device 30 converts the recorded video into a video (normalized video) in which the camera moves at a constant speed, and edits the normalized video. It is preferable to generate the movement route image by In order for the image providing device 30 to create a normalized moving image, the moving speed of the camera must be included in the meta information of the recorded image.

The recorded video captured in the past along the movement route between the first exhibition area and the second exhibition area is, for example, the video captured by the camera 23 after moving the robot 20A or 20B outside the exhibition area. should be As a result, the connection between the real-time image in the exhibition implementation area and the moving route image generated by the image providing device 30 becomes smooth. In addition, recorded images taken in the past on the route between the first exhibition area and the second exhibition area may be captured by cameras mounted on smartphones, drive recorders, etc., fixed points installed on the route. It may be an image captured by a camera or the like.

The video display device 10 changes the playback speed of the movement route video by transmitting instruction information regarding the playback speed input to the input unit 13 to the video providing device 30 . For example, by increasing the playback speed of the moving route video, it is possible to shorten the time required for pseudo-moving between the first exhibition area and the second exhibition area.

In the next step S70, the image providing device 30 determines whether or not the user enters the exhibition area in a simulated manner. For example, when the viewpoint of the moving route image reaches the entrance of the exhibition area, the image providing device 30 may determine that the user will enter the exhibition area in a simulated manner.

If it is determined in step S70 that the user will not enter in a pseudo manner, the process of step S60 is continued. On the other hand, if it is determined in step S70 that the user will enter in a pseudo manner, the process proceeds to step S20.

By the operation described above, the display control unit 11 of the video display device 10 controls the real-time video in the first exhibition area or the second exhibition area photographed by the camera 23, the first exhibition area, and the second exhibition area. and the moving route video based on the recorded video taken in the past on the moving route between and is displayed on the display unit 12. - 特許庁

For example, the user simulates looking around the first exhibition area, then moves from the first exhibition area to the second exhibition area, and finally looks at the second exhibition area. When rotating, the video displayed by the display unit 12 transitions in order of the real-time video 41 in the first exhibition implementation area, the moving route video 42, and the real-time video 43 in the second exhibition implementation area, as shown in FIG. .

This video switching can give the user both a sense of realism related to the exhibition and a sense of reality related to movement between exhibition implementation areas.

In addition, since the moving route video is generated based on the recorded video, even if the moving route is congested in real time, the moving route video will not be adversely affected.

In addition, since the user moves between the first exhibition area and the second exhibition area based on the moving route image, the scenery during movement, the shops along the moving route, and the shops along the moving route It can be expected that serendipity will occur with respect to the products that are lined up.

In step S30, the real-time image captured by the camera 23 is displayed on the display unit 12, but there are cases where it is desired to simulate the exhibit from an angle that cannot be captured by the camera 23 (for example, directly above the exhibit). . Therefore, in step S30, a real-time image captured by the camera 23 and another camera different from the camera 23 in the exhibition area (for example, a fixed point camera installed in the exhibition area, a camera mounted on a drone, etc.) ) may be displayed on the display unit 12 by switching to the in-area image based on the recorded image taken in the past. The display control unit 11 may switch between the real-time image and the in-area image based on the instruction information input to the input unit 13 .

In step S60, the robot 20B, which is not linked to another video display device 10, is preliminarily (from the moving route video) to a point where it is expected to switch from the moving route video to the real-time video in the second exhibition implementation area. (before switching to the real-time video in the second exhibition implementation area). As a result, it is possible to switch from the moving route image to the real-time image of the second exhibition area without giving the user a sense of discomfort.

The video providing device 30 may predict the point at which the moving route video is switched to the real-time video within the second exhibition area, for example, based on the distance between the viewpoint of the moving route video and the entrance of the second exhibition area. Specifically, when the distance between the viewpoint of the movement route image and the entrance of the second exhibition implementation area approaches a certain distance L1, the image providing device 30 moves the entrance from the movement route image to the second exhibition. It is good to expect that it is the point where the real-time video in the area switches to.

For example, as shown in FIG. 5, the second exhibition area 51 has a first entrance 52 and a second entrance 53, and the distance between the viewpoint 54 of the moving route image and the first entrance 52 is a constant distance L1. , and when the distance between the viewpoint 54 of the moving route image and the second entrance 53 is a distance L2 longer than the fixed distance L1, the image providing device 30 moves the first entrance 52 from the moving route image to the second exhibition area. The robot 20B is directed to the first entrance 52, anticipating that it will be a point where the real-time image will switch to the inside.

As another method, in step S60, the video providing device 30 may set the stop position of the robot 20B as the point at which the moving route video is switched to the real-time video within the second exhibition implementation area. As a result, it is possible to switch from the moving route image to the real-time image of the second exhibition area without giving the user a sense of discomfort.

For example, as shown in FIG. 6, when the second exhibition area 51 has a first entrance 52 and a second entrance 53 and the robot 20B is stopped at the second entrance 53, the moving route image The image providing apparatus 30 may forcibly set the movement route so that the viewpoint 54 moves to the second entrance 53, and the end point of the movement route is set to the second point according to the instruction information input to the input unit 13 by the user. An arrow or the like may be superimposed on the moving route image so that the second entrance 53 is displayed, and the end point of the moving route may be guided to the second entrance 53 .

<Others>
The above embodiments should be considered illustrative in all respects and not restrictive, and the technical scope of the present invention is indicated by the scope of claims rather than the description of the above embodiments. It should be understood that all modifications that fall within the meaning and range of equivalents of the claims are included.

For example, in the above-described embodiments, the robot was a self-propelled robot, but it may be a drone or the like. In addition, instead of the robot, a vehicle running on the track was used, and a camera capable of shooting 360° was mounted on the vehicle running on the track, and a part of the image captured by the camera capable of shooting 360° was cut out. The image may be adopted as a real-time image, and the clipping position thereof may be remotely controlled.

In addition, the image display system may be configured to require charging for simulated entry into each exhibition or event implementation area, and charging is required only for simulated entry into one exposition, theme park, or the like. may be configured to require

1 video display system 10 video display device 11 display control section 12 display section 13 input section 14 communication section 15 speaker 16 microphone 20A, 20B robot 21 control section 22 drive section 23 camera 24 communication section 25 microphone 26 speaker 30 video providing device 31 control Section 32 Recording Section 33 Communication Section 41 Real-time image of the first exhibition area 42 Movement route image 43 Real-time image of the second exhibition area 51 Second exhibition area 52 First entrance 53 Second entrance 54 Movement route picture

Claims (9)

a display unit;
A display control unit that controls the display unit,
The display control unit moves based on a real-time image within the viewing area captured by a camera mounted on a moving object moving within the viewing area and a recorded image captured in the past on a moving route between the viewing areas. An image display device that switches between route images and displays them on the display unit. 2. The image display device according to claim 1, wherein the reproduction speed of said movement route image is variable. The display control unit switches between the real-time video, the moving route video, and the area video based on the video recorded in the past by a camera different from the camera in the viewing area, and displays the video on the display unit. 3. The image display device according to claim 1 or 2, for displaying. A video display device according to any one of claims 1 to 3;
the moving body;
an image providing device for providing the real-time image and the movement route image to the image display device;
A video display system. 5. The image display system according to claim 4, wherein said mobile body is a remotely controllable robot. 6. The video display system according to claim 5, wherein said robot moves in advance to a point where said moving route video is expected to switch to said real-time video. 7. The video display system according to claim 6, wherein a point at which the moving route video is switched to the real-time video is set to a stop position of the robot. the moving body is movable outside the viewing area;
8. The image display system according to any one of claims 4 to 7, wherein said movement route image is generated based on a recorded image previously taken by said camera on a movement route between said viewing areas. Switching between a real-time image of the viewing area captured by a camera mounted on a moving body moving within the viewing area and a movement route image based on recorded images previously captured along the movement route between the viewing areas. Display, image display method.

Images