JP2022179324A - Display processing apparatus, display processing method, program, and recording medium - Google Patents

Abstract

To provide a display processing apparatus capable of enhancing realism of virtual travel.SOLUTION: In a display processing apparatus 10 of the present invention, a background image acquisition unit 11 acquires a background image related to virtual travel, a display area setting unit 12 sets a display area for displaying participants of the virtual travel in the background image for each of the participants, a video acquisition unit 13 acquires a video related to the participants, and a display processing unit 14 changes the background of the participants in the video to a partial image of the background image within the display area and displays the video of all the participants on the same screen.SELECTED DRAWING: Figure 1

Description

The present invention relates to a display processing device, a display processing method, a program, and a recording medium.

Virtual travel (also referred to as a virtual tour or the like) is available as a technology that allows users to enjoy the feeling of travel while staying at home. Virtual travel is a technology that provides tourists with videos and images of tourist spots around the world using the Internet (for example, Patent Document 1).

In virtual tours, there is also a tour in which terminals between a guide and a tourist communicate in real time, and the guide and the tourist see each other's faces by sending and receiving self-portrait videos.

JP 2007-156562 A

However, in a virtual trip involving transmission and reception of self-portrait moving images, for example, a part of the inside of the home or office is reflected in the self-portrait moving images, so there is a problem that the sense of presence is lost. In addition, there is a demand for improving the sense of realism in virtual travel, not limited to virtual travel involving transmission and reception of self-portrait moving images.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a display processing device, a display processing method, a program, and a recording medium that can enhance the presence of a virtual trip.

In order to achieve the above object, the display processing device of the present invention includes:
including a background image acquisition unit, a display area setting unit, a video acquisition unit, and a display processing unit,
The background image acquisition unit acquires a background image related to the virtual trip,
The display area setting unit sets, for each participant, a display area for displaying the virtual travel participant in the background image,
The moving image acquisition unit acquires a moving image related to the participant,
The display processing unit changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. is a device for performing

The display processing method of the present invention includes:
including a background image acquisition step, a display area setting step, a video acquisition step, and a display processing step;
The background image obtaining step obtains a background image related to the virtual travel,
The display area setting step sets, for each participant, a display area for displaying the participant of the virtual tour in the background image,
The moving image acquisition step acquires a moving image of the participant,
The display processing step changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. is a method of executing

According to the present invention, it is possible to enhance the sense of presence during participation in a virtual trip.

FIG. 1 is a block diagram showing an example configuration of a display processing device according to a first embodiment. FIG. 2 is a block diagram illustrating an example of the hardware configuration of the display processing device according to the first embodiment; 3 is a flowchart illustrating an example of processing in the display processing device according to the first embodiment; FIG. FIG. 4 is a schematic diagram showing an example of a screen displayed on a participant's terminal.

In the display processing device of the present invention, for example,
The display processing unit further displays the background image on the same screen, and displays the video of the participant whose background has been changed in the display area on the background image. good too.

The display processing device of the present invention, for example,
Furthermore, including a screen imaging unit,
The screen imaging unit may capture a screen displayed on the terminal of the participant by the display processing unit.

In the display processing device of the present invention, for example,
The moving image about the participant may be at least one of a moving image including the face of the participant and a moving image including an avatar representing the participant.

In the display processing device of the present invention, for example,
The virtual trip may be a virtual trip held in at least one of the real space and the virtual space.

In the display processing method of the present invention, for example,
In the display processing step, the background image is further displayed on the same screen, and the video of the participant whose background is changed is displayed in the display area on the background image. good too.

The display processing method of the present invention is, for example,
Furthermore, including a screen imaging process,
The screen capturing step may capture a screen displayed on the participant's terminal by the display processing step.

In the display processing method of the present invention, for example,
The moving image about the participant may be at least one of a moving image including the face of the participant and a moving image including an avatar representing the participant.

In the display processing method of the present invention, for example,
The virtual trip may be a virtual trip held in at least one of the real space and the virtual space.

The program of the present invention is a program for causing a computer to execute each step of the method of the present invention as a procedure.

A recording medium of the present invention is a computer-readable recording medium recording the program of the present invention.

In the present invention, "virtual travel" is technology that provides tourists with a virtual travel experience using the Internet. The virtual tour is also called, for example, a virtual tour, a pseudo-tour, a remote sightseeing, an online tour, or the like. A virtual tour is performed, for example, by real-time communication between a guide's terminal and a tourist's terminal. The guide, for example, distributes videos and images of tourist spots around the world to the tourists, and guides the tourist spots. This allows the tourist to experience a virtual trip. The virtual tour, for example, can be said to be a technology that is held on a preset date and time and provides tourists with an online travel experience. The moving image and image of the tourist spot are not particularly limited as long as they are related to the tourist spot, and may be, for example, a 360-degree moving image (a moving image capturing all 360 degrees).

In the present invention, the term "participant" is not particularly limited as long as it is a person participating in the virtual tour, and includes, for example, guides and tourists.

Next, an embodiment of the present invention will be described with reference to the drawings. The invention is not limited to the following embodiments. In each figure below, the same reference numerals are given to the same parts. In addition, the description of each embodiment can be used with reference to each other's description unless otherwise specified, and the configurations of the respective embodiments can be combined unless otherwise specified.

[Embodiment 1]
FIG. 1 is a block diagram showing the configuration of an example of the display processing device 10 of this embodiment. As shown in FIG. 1, the device 10 includes a background image acquisition section 11, a display area setting section 12, a video acquisition section 13, and a display processing section . Further, the device 10 may further include a screen imaging unit 15 as an arbitrary configuration, for example. The device 10 may be, for example, a device capable of providing virtual travel to participants or a device included in such a device.

The device 10 may be, for example, a single device including each of the above units, or may be a device in which each of the above units can be connected via a communication network. Further, the device 10 can be connected to an external device, which will be described later, via the communication network. The communication line network is not particularly limited, and a known network can be used, and may be wired or wireless, for example. The communication network includes, for example, the Internet line, WWW (World Wide Web), telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (local 5G), and the like. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be direct communication between devices (Ad Hoc communication), infrastructure communication, or indirect communication via an access point. The device 10 may be incorporated in a server as a system, for example. Further, the device 10 may be, for example, a personal computer (PC, for example, desktop type, notebook type), a smart phone, a tablet terminal, etc. in which the program of the present invention is installed. Furthermore, the device 10 may be in a form of cloud computing, edge computing, or the like, in which at least one of the units is on the server and the other units are on the terminal, for example.

FIG. 2 illustrates a block diagram of the hardware configuration of the device 10. As shown in FIG. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, an output device 106, a communication device 107, and the like. Each unit of the device 10 is interconnected via a bus 103 by each interface (I/F). Note that these are only examples, and the hardware configuration of the device 10 is not limited to this as long as the processing of each section can be executed. Also, the number of central processing units 101 and the like included in the apparatus 10 is not limited to the example shown in FIG.

A central processing unit 101 is responsible for overall control of the apparatus 10 . In the apparatus 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and reads and writes various types of information. Specifically, for example, the central processing unit 101 functions as a background image acquisition unit 11 , a display area setting unit 12 , a moving image acquisition unit 13 , a display processing unit 14 and a screen imaging unit 15 .

Bus 103 can also be connected to external devices, for example. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, and the like. For example, the device 10 can be connected to an external network (the above-mentioned communication line network) by means of a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 is, for example, a main memory (main storage device). When the central processing unit 101 performs processing, for example, the memory 102 reads various operating programs such as the program of the present invention stored in the storage device 104 to be described later, and the central processing unit 101 reads from the memory 102 Get the data and run the program. The main memory is, for example, RAM (random access memory). Also, the memory 102 may be, for example, a ROM (read only memory).

The storage device 104 is also called a so-called auxiliary storage device, for example, in contrast to the main memory (main storage device). As described above, the storage device 104 stores operating programs including the program of the present invention. Storage device 104 may be, for example, a combination of a recording medium and a drive that reads from and writes to the recording medium. The recording medium is not particularly limited, and may be, for example, a built-in type or an external type, and includes HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, and the like. be done. The storage device 104 may be, for example, a hard disk drive (HDD) in which a recording medium and drive are integrated, and a solid state drive (SSD).

In the device 10, the memory 102 and the storage device 104 store log information, information acquired from an external database (not shown) or an external device, information generated by the device 10, and data stored when the device 10 executes processing. It is also possible to store various information such as information to be used. At least part of the information may be stored in an external server other than the memory 102 and the storage device 104, or may be distributed and stored in a plurality of terminals using blockchain technology or the like. .

The device 10 may further include user interfaces such as an input device 105 and an output device 106, for example. The input device 105 is, for example, a device for inputting characters, numbers, positions of objects displayed on the screen, images, sounds, and the like. devices, microphones, sensors, and the like. Examples of the output device 106 include a display device (LED display, liquid crystal display), printer, speaker, and the like. The input device 105 and the output device 106 may be, for example, a user interface such as a touch panel that combines them.

Next, an example of the display processing method of this embodiment will be described based on the flowchart of FIG. The display processing method of this embodiment is implemented as follows, for example, using the display processing device 10 of FIG. Note that the display processing method of this embodiment is not limited to the use of the display processing device 10 of FIG. Also, the steps shown in parentheses in FIG. 3 are optional steps.

In the display processing method of the present embodiment, the process executed by the background image acquisition unit 11 is the background image acquisition process, the process executed by the display area setting unit 12 is the display area setting process, and the video acquisition unit 13 The process executed is the video acquisition process, the process executed by the display processing unit 14 is the display processing process, and the process executed by the screen imaging unit 15 is the screen imaging process.

First, the background image acquisition unit 11 acquires a background image related to the virtual trip (S11). The background image is not particularly limited as long as it is an image related to the content of the virtual trip. Examples of the background image include an image of a tourist attraction, an image of a sightseeing bus, an image of a spot explained in the virtual trip, and the like. The background image is an image that is shared among participants, which will be described later. Also, there may be a plurality of background images for one virtual trip, for example. The method of acquiring the background image is not particularly limited. For example, the background image may be acquired from an external device via the communication network, or may be stored in the memory 102 and the storage device 104 in advance. The background image may be obtained by reading out the background image, or may be obtained by capturing an image with an imaging device that is the input device 105 .

The virtual trip may be held in real space, virtual space, or the like, for example. The virtual trip held in the real world is, for example, a travel experience completed on a terminal. More specifically, for example, there is a travel experience in which a guide distributes videos and images on a terminal and tourists view the distributed videos and images. The virtual trip held in the virtual space includes, for example, a trip held in VR (virtual reality), metaverse, or other space.

Next, the display area setting unit 12 sets a display area for displaying the participants of the virtual tour in the background image for each of the participants (S12). That is, a unique display area is set for each participant. The display area setting unit 12 may associate, for example, the identification information for the terminal of the participant with the display area. The display area setting unit 12 can set any area as the display area within the background image. For example, the display area setting unit 12 may set a plurality of display areas for one background image. Further, the display area setting unit 12 may set the same number of display areas as the number of participants (the number of terminals of the participants, that is, the number of the identification information), for example. The size, shape, etc. of the display area are not particularly limited. For example, the display area setting unit 12 may or may not cut an arbitrarily set range from the background image as the display area. As will be described later, an image within the range set as the display area is called a partial image. Further, the display area setting unit 12 may, for example, associate one participant (one terminal) with one display area (one terminal) of the participant (the participant's terminal). On the other hand, two or more (two or more terminals) participants (terminals of the participants) may be linked. The participants (terminals of the participants) may be managed in advance by, for example, identification information (ID (identifier), etc.). The identification information may be acquired from an external device, or may be stored in advance, for example.

Next, the moving image acquisition unit 13 acquires a moving image of the participant (S13). For example, the moving image acquisition unit 13 acquires the moving image from the terminal of the participant via the communication line network.

The moving image about the participant is, for example, a moving image including the face of the participant, a moving image including an avatar representing the participant, or the like. The avatar may be, for example, an avatar capable of tracing the face of the participant and expressing the facial expression of the participant in real time or non-real time. By using avatars capable of expressing the facial expressions of the participants, it is possible to grasp the reactions of the participants even in a virtual trip held in a virtual space such as the Metaverse. On the other hand, the avatar may be, for example, an avatar without trace of the participant's face, an avatar that does not express the participant's facial expressions. The moving image including the avatar may be, for example, a moving image including the face of the avatar corresponding to the face of the participant, or a moving image including the whole body of the avatar corresponding to the whole body of the participant. The tracing of the participant's face and the avatarization of the participant may be executed when the moving image is acquired by the moving image acquisition unit 13, or the participant's terminal and the participant's terminal and this apparatus may be executed when the moving image is acquired by the moving image acquisition unit 13. 10 may also be performed by other devices that intervene with 10 . Using the avatar can improve the anonymity of the participant, for example. The moving image can also be referred to as, for example, a self-portrait moving image, a live moving image, or the like. The live video is, for example, a video captured (and avatarized) of the participant in parallel with the provision of the virtual tour. The moving image may be, for example, a moving image with sound.

The virtual tour may be provided (distributed) by the device 10 or may be provided (distributed) by an external device, for example. A technique for providing the virtual tour is not particularly limited.

Then, the display processing unit 14 changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. (S14) and ends (END). Specifically, the display processing unit 14, for example, extracts a body image, which is a portion in which the body (face, etc.) of the participant is shown, from the moving image, and superimposes the body image on the partial image. The extraction method is not particularly limited, and a known algorithm can be used. The processing may be, for example, processing for the screen of the display device of the terminal of the participant, or may be processing for the screen of the display device, which is the output device 106 . It should be noted that the image of the portion other than the partial image in the background image does not have to be displayed on the same screen, and may be displayed as described later. When the image of the other part is not displayed on the same screen, the display processing unit 14, for example, cuts the partial image from the background image and superimposes the body image on the cut partial image.

For example, in the step (S14), the display processing unit 14 further displays the background image on the same screen, and displays the moving image of the participant whose background has been changed in the display area within the background image. may be displayed. In this case, for example, the display processing unit 14 superimposes the body image on the partial image without cutting out the partial image from the background image. According to this, it is possible to produce a sense of unity between the inside of the moving image and the outside of the moving image, and it is possible to further enhance the presence.

Furthermore, the device 10 may further include the screen imaging unit 15, for example, as described above. For example, after the step (S14), the image capturing unit 15 captures the screen displayed on the terminal of the participant by the display processing unit 14 (S15). The imaging of the screen is also called a screen shot, for example. As a result, for example, a commemorative photo, a group photo, or the like can be taken during the virtual trip. Until now, it has been difficult to take a commemorative photo or the like in a virtual trip held in a virtual space such as the Metaverse, since each participant can act freely. On the other hand, according to the invention according to the present embodiment, since the moving images of all the participants are displayed on the same screen, even if it is a virtual trip held in the virtual space, can be easily done.

FIG. 4 shows an example of a screen displayed on a participant's terminal. Although the number of participants is three in FIG. 4, it is not limited to this. The background image acquisition unit 11 acquires an image showing a mountain, which is the stage of the virtual trip, as a background image related to the virtual trip. The display area setting unit 12 sets the same number of display areas 1 as the number of participants in the background image, as described above. Then, as shown in the figure, the display processing unit 14 displays the moving image about the participant acquired by the moving image acquisition unit 13 in the display area 1, and replaces the background in the moving image with the background image corresponding to the position of the display area 1. (said partial image). In FIG. 4, the display area 1 is indicated by a frame line so that the position of the display area 1 can be easily understood, but the frame line does not have to be displayed on the screen.

According to the present embodiment, by using the background image common among the participants and displaying the moving images about the participants in the display area, a sense of unity is generated between the moving images, and the realism of the virtual trip is enhanced. can heighten your senses. When one person (one terminal) of the participant (the participant's terminal) is simply displayed in association with one of the display areas, the background of each of the participants will be different. Therefore, in the invention according to this embodiment, since the background image is used in common among the participants, a sense of unity is created between the moving images. Similarly, when two or more participants (two or more terminals) of the participants (the terminals of the participants) are associated with one display area, a sense of unity is generated between the moving images.

[Embodiment 2]
The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, "procedure" may be read as "processing". Moreover, the program of this embodiment may be recorded on a computer-readable recording medium, for example. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include read-only memory (ROM), hard disk (HD), optical disc, and the like.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes can be made to the configuration and details of the present invention within the scope of the present invention that can be understood by those skilled in the art.

This application claims priority based on Japanese Patent Application No. 2021-086139 filed on May 21, 2021, and the entire disclosure thereof is incorporated herein.

<Appendix>
Some or all of the above-described embodiments can be described as in the following appendices, but are not limited to the following.
(Appendix 1)
including a background image acquisition unit, a display area setting unit, a video acquisition unit, and a display processing unit,
The background image acquisition unit acquires a background image related to the virtual trip,
The display area setting unit sets, for each participant, a display area for displaying the virtual travel participant in the background image,
The moving image acquisition unit acquires a moving image related to the participant,
The display processing unit changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. A display processor that executes
(Appendix 2)
The display according to Supplementary Note 1, wherein the display processing unit further displays the background image on the same screen, and displays the video of the participant who changed the background in the display area on the background image. processing equipment.
(Appendix 3)
Furthermore, including a screen imaging unit,
3. The display processing device according to appendix 1 or 2, wherein the screen imaging unit captures a screen displayed on the terminal of the participant by the display processing unit.
(Appendix 4)
4. The display processing device according to any one of appendices 1 to 3, wherein the video about the participant is at least one of a video including the face of the participant and a video including an avatar representing the participant.
(Appendix 5)
5. The display processing device according to any one of appendices 1 to 4, wherein the virtual trip is a virtual trip held in at least one of a real space and a virtual space.
(Appendix 6)
including a background image acquisition step, a display area setting step, a video acquisition step, and a display processing step;
The background image obtaining step obtains a background image related to the virtual travel,
The display area setting step sets, for each participant, a display area for displaying the participant of the virtual tour in the background image,
The moving image acquisition step acquires a moving image of the participant,
The display processing step changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. , the display processing method.
(Appendix 7)
The display according to attachment 6, wherein the display processing step further displays the background image on the same screen, and displays the video of the participant whose background has been changed in the display area on the background image. Processing method.
(Appendix 8)
Furthermore, including a screen imaging process,
8. The display processing method according to appendix 6 or 7, wherein the screen capturing step captures a screen displayed on the participant's terminal by the display processing step.
(Appendix 9)
9. The display processing method according to any one of appendices 6 to 8, wherein the video about the participant is at least one of a video including the face of the participant and a video including an avatar representing the participant.
(Appendix 10)
10. The display processing method according to any one of appendices 6 to 9, wherein the virtual trip is a virtual trip held in at least one of a real space and a virtual space.
(Appendix 11)
Including a background image acquisition procedure, a display area setting procedure, a video acquisition procedure, and a display processing procedure,
The background image obtaining step obtains a background image related to the virtual trip,
The display area setting step sets, for each participant, a display area for displaying the participant of the virtual trip in the background image,
The video acquisition step acquires a video about the participant,
The display processing procedure changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. A program that causes a computer to do something.
(Appendix 12)
12. The program according to attachment 11, wherein the display processing procedure further displays the background image on the same screen, and displays the moving image of the participant whose background has been changed in the display area on the background image. .
(Appendix 13)
further comprising a screen capture procedure;
13. The program according to appendix 11 or 12, wherein the screen capturing procedure captures a screen displayed on the participant's terminal by the display processing procedure.
(Appendix 14)
14. The program according to any one of appendices 11 to 13, wherein the video about the participant is at least one of a video including the face of the participant and a video including an avatar representing the participant.
(Appendix 15)
15. The program according to any one of appendices 11 to 14, wherein the virtual trip is a virtual trip held in at least one of real space and virtual space.
(Appendix 16)
A computer-readable recording medium recording the program according to any one of appendices 11 to 15.

According to the present invention, it is possible to enhance the sense of presence of a virtual trip. Therefore, the present invention is particularly useful, for example, when holding a virtual tour in real space or virtual space.

10 Display processing unit 11 Background image acquisition unit 12 Display area setting unit 13 Moving image acquisition unit 14 Display processing unit 15 Screen imaging unit 101 Central processing unit 102 Memory 103 Bus 104 Storage device 105 Input device 106 Output device 107 Communication device

Claims (10)

including a background image acquisition unit, a display area setting unit, a video acquisition unit, and a display processing unit,
The background image acquisition unit acquires a background image related to the virtual trip,
The display area setting unit sets, for each participant, a display area for displaying the virtual travel participant in the background image,
The moving image acquisition unit acquires a moving image related to the participant,
The display processing unit changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. A display processor that executes 2. The display processing unit according to claim 1, wherein the background image is further displayed on the same screen, and the video of the participant whose background is changed is displayed in the display area on the background image. Display processor. Furthermore, including a screen imaging unit,
3. The display processing device according to claim 1, wherein said screen imaging unit images a screen displayed on said participant's terminal by said display processing unit. The display processing device according to any one of claims 1 to 3, wherein the moving image about the participant is at least one of a moving image including the face of the participant and a moving image including an avatar representing the participant. The display processing device according to any one of claims 1 to 4, wherein the virtual trip is a virtual trip held in at least one of a real space and a virtual space. including a background image acquisition step, a display area setting step, a video acquisition step, and a display processing step;
The background image obtaining step obtains a background image related to the virtual travel,
The display area setting step sets, for each participant, a display area for displaying the participant of the virtual tour in the background image,
The moving image acquisition step acquires a moving image of the participant,
The display processing step changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. , the display processing method. 7. The display processing step according to claim 6, further comprising displaying the background image on the same screen, and displaying the video of the participant who changed the background in the display area on the background image. Display processing method. Furthermore, including a screen imaging process,
8. The display processing method according to claim 6, wherein said screen image capturing step captures a screen displayed on said participant's terminal by said display processing step. 9. The display processing method according to any one of claims 6 to 8, wherein the video about the participant is at least one of a video including the face of the participant and a video including an avatar representing the participant. Including a background image acquisition procedure, a display area setting procedure, a video acquisition procedure, and a display processing procedure,
The background image obtaining step obtains a background image related to the virtual trip,
The display area setting step sets, for each participant, a display area for displaying the participant of the virtual trip in the background image,
The video acquisition procedure acquires a video about the participant,
The display processing procedure changes the background of the participants in the moving image to a partial image of the background image within the display area, and displays the moving images of all the participants on the same screen. A program that causes a computer to do something.

Images