WO2022259480A1

WO2022259480A1 - Video processing device, video processing method, and video processing program

Info

Publication number: WO2022259480A1
Application number: PCT/JP2021/022173
Authority: WO
Inventors: 弘員柿沼; 誉宗巻口; 秀信長田
Original assignee: 日本電信電話株式会社
Priority date: 2021-06-10
Filing date: 2021-06-10
Publication date: 2022-12-15
Also published as: JPWO2022259480A1

Abstract

A video processing device (20) comprises a reception unit (2011), a distance inference unit (2012), a video processing unit (2014), and a transmission unit (2015). The reception unit receives a video. The distance inference unit infers a viewing distance on the assumption that a person was viewing a landscape in the range equivalent to that of the video. The video processing unit processes the video so that the size of a reference object appearing on the video at the viewing distance conforms to the actual size of the reference object. The transmission unit transmits the processed video to a display device.

Description

VIDEO PROCESSING DEVICE, VIDEO PROCESSING METHOD, AND VIDEO PROCESSING PROGRAM

The embodiments relate to a video processing device, a video processing method, and a video processing program.

Users may find it difficult to feel a sense of unity with the local area when watching events such as music and theater performances or watching sports through images displayed on a display such as a television (television). be. One reason for this is, for example, that the presence of spectators around the user cannot be felt.

On the other hand, Non-Patent Document 1 and Non-Patent Document 2 propose a technology that makes the user feel the presence of others by presenting the appearance and remarks of the audience on the screen.

Another reason why it is difficult for users to feel a sense of unity with the site is that images are displayed on displays such as televisions with a different sense of distance from reality. As a result, even if the appearance of the audience is presented, the user will feel a sense of incongruity, and it is difficult for the user to feel a sense of immersion as if they were in the actual venue.

An embodiment is a video processing device that allows a user to feel as if they were watching at an actual venue when watching events such as music and theater or watching sports through video in a remote location. , provides a video processing method and a video processing program.

The video processing device of the embodiment has a receiving unit, a distance estimating unit, a video processing unit, and a transmitting unit. The receiving unit receives video. A distance estimating unit estimates a viewing distance when it is assumed that a human being has viewed scenery in the same range as the image. The image processing unit processes the image so that the size of the reference object appearing in the image at the viewing distance matches the actual size of the reference object. The transmission unit transmits the processed video to the display.

According to the embodiment, when viewing events such as music and theater, or watching sports through video in a remote location, the user can feel as if he/she is watching at an actual venue. A processing device, a video processing method, and a video processing program are provided.

FIG. 1 is a diagram showing a schematic configuration of a video distribution system including a video processing device according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a video processing device; FIG. 3 is a functional block diagram of the video processing device. FIG. 4 is a flow chart showing the operation of the video processing device. FIG. 5 is a diagram showing the relationship between the height of the performer and the shooting range of the camera in the height direction. FIG. 6 is a diagram showing the distance relationship between the user and the performer when it is assumed that the user is watching the performer at the venue. FIG. 7A is a diagram showing a schematic configuration of a video distribution system using a transmissive wearable display. FIG. 7B is a diagram showing a schematic configuration of a video distribution system using a transmissive wearable display.

Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 is a diagram showing a schematic configuration of a video distribution system including a video processing device according to an embodiment. The video distribution system 1 has a camera 10 , a video processing device 20 and a display 30 .

The cameras 10 are installed, for example, in the audience seats AS at venues where events such as music and theater are held. Specifically, the audience seats AS are arranged to face the stage S, for example. A space for installing the camera 10 is provided in a part of the audience seats AS. The camera 10 may be a camera configured to be movable or may be a fixed position camera. The camera 10 shoots the performer P at a predetermined frame rate and generates image data of the performer P. The video data may be resized according to the size of the video that can be displayed by the display 30, or the like. For example, if the size of an image that can be displayed by the display 30 is a full HD (High Definition) size, the image data may be resized to 1920 pixels×1080 pixels. Also, the camera 10 is connected so as to be able to communicate with the video processing device 20 . Video data captured by the camera 10 is transmitted to the video processing device 20 .

Here, the performer P in the embodiment is a general term for people who perform various activities in an event, such as a performer or the like for a music event, or an actor or the like for a theater event. In embodiments, the performer P is not limited to a person performing a specific expressive activity.

Also, in FIG. 1, the number of cameras 10 is one. However, the number of cameras 10 is not limited to one. For example, the cameras 10 may be installed at multiple positions within the hall.

The video processing device 20 processes the video transmitted from the camera 10. For example, the video processing device 20 determines that the distance from the user U to the virtual display surface arranged in the three-dimensional space and the height of the performer P on the virtual display surface perceived by the user U through the display on the display 30 are different from the actual height of the performer P on the virtual display surface. The image is processed so that it is the distance and height that you would perceive if you were looking at the performer P at the venue. The video processing device 20 then transmits the processed video to the display 30 . The video processing device 20 may be installed inside the venue or outside the venue. As in the venue, the video processing device 20 may be included in the camera 10 . Also, outside the venue, the video processing device 20 may be included in the display 30 . Of course, the video processing device 20 may be separate from the camera 10 and the display 30 . Details of the video processing device 20 will be described later.

The display 30 is configured to be able to communicate with the video processing device 20, and displays the video transmitted from the video processing device 20. The display 30 is, for example, a non-transmissive glasses-type wearable display that is worn on the head of the user U who is in a remote location with respect to the venue. The display 30 is configured to allow three-dimensional display. For example, the display 30 has a display unit at each eye position, and displays an image obtained by synthesizing the image of the camera 10 transmitted from the image processing device 20 and the avatar image of the audience on each display unit. Thus, the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience are perceived at a position at a predetermined viewing distance from the user U. Here, the avatar is an image imitating the audience, and may be, for example, a two-dimensional or three-dimensional illustration image representing the audience. Also, the avatar may be a CG image or the like using actual photography. A configuration for three-dimensional display by the display 30 is not limited to a specific configuration. Also, the location of the user U may be a remote location from the venue, such as the home of the user U or a public viewing venue provided separately from the venue.

In addition to the camera 10, the venue may also have a camera 40 and a large service monitor 50. Camera 40 is communicatively connected to large service monitor 50 . The camera 40 is arranged on the stage S, for example, and configured to photograph the performer P on the stage S. The number of cameras 40 is not limited to one. For example, the camera 40 may also be installed in the auditorium AS. The large service monitor 50 is installed, for example, on the stage S so as to face the audience seats AS, and displays the image captured by the camera 40 on a large screen.

In the video distribution system 1 shown in FIG. 1, for example, a spectator A1 sitting on the audience seat AS near the stage S and a spectator A2 sitting on the audience seat AS facing the performer P see the performer P on the stage S with the naked eye. can be seen in However, the audience A1 near the stage S and the audience A2 far from the stage S perceive different scenery. Specifically, the performer P is smaller in the scenery v2 perceived by the audience A2 than in the scenery v1 perceived by the audience A1. A spectator A3 sitting in the audience seat AS far from the stage S can see the performer P with the naked eye through the image on the large service monitor 50. FIG. Therefore, spectator A3 also perceives scenery v3 different from that of spectators A1 and A2. In this way, people perceive different scenery depending on the viewing position even if they see the performer P at the same venue. This is one of the reasons why the user U feels uncomfortable when viewing the video captured by the camera 10 at a remote location.

FIG. 2 is a diagram showing an example of the hardware configuration of the video processing device 20. As shown in FIG. The video processing device 20 may be configured as a computer. The video processing device 20 does not have to be a single computer, and may be composed of a plurality of computers. As shown in FIG. 2, the video processing device 20 has a processor 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a storage 204, an input device 205, and a communication module 206. is doing. Here, the video processing device 20 may further have a display or the like.

The processor 201 is a processing circuit capable of executing various programs and controls the overall operation of the video processing device 20 . The processor 201 may be a processor such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit). Also, the processor 201 may be an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. Furthermore, the processor 201 may be composed of a single CPU or the like, or may be composed of a plurality of CPUs or the like.

The ROM 202 is a non-volatile semiconductor memory and holds programs and control data for controlling the video processing device 20 .

The RAM 203 is, for example, a volatile semiconductor memory, and is used as a work area for the processor 201.

The storage 204 is a nonvolatile storage device such as a hard disk drive (HDD) or solid state drive (SSD). The storage 204 holds programs 2041 , variables 2042 and spectator data 2043 .

The program 2041 is a program for processing the video of the camera 10. The program 2041 includes a process of estimating the shooting range from the image of the camera 10, a process of estimating a virtual viewing distance when it is assumed that a person sees a scene in the same range as the estimated shooting range, and a process of estimating the estimated viewing distance. It is a program for causing the processor 201 to execute a process of processing an image based on the distance.

Variables 2042 are various variables used for video processing. In an embodiment, variables 2042 include the length of the fiducial, the vertical size of the image sensor, the focal length, and the number of vertical pixels of the image.

The length of the reference object is the actual length of the reference object used to estimate the shooting range. The reference object may be any object with a known length that can appear in the image captured by the camera 10 . For example, performer P may be a reference. When the reference object is performer P, the length of the reference object may be performer P's height. For example, the height of the performer P may be input to the video processing device 20 by the organizer of the event or the like before the event is held.

The vertical size of the image sensor is the vertical size when the retina of the human eye is regarded as the image sensor of the camera. For example, if the retina of the human eye is equated with a full size image sensor, the vertical size of the image sensor is 24 mm. If the retina function of the human eye is considered equivalent to that of an APS-C size image sensor, the vertical size of the image sensor is 16.7 mm.

The focal length is the value of the focal length when the lens of the human eye is regarded as a camera lens. Generally speaking, when the human eye is regarded as a camera consisting of a lens and a full-size image sensor, its focal length is said to be equivalent to 10 to 12 mm. However, in reality, humans cannot process all the light that enters the eye from a field of view equivalent to a focal length of 10 to 12 mm, and only processes the light that enters the eye from a partial range of the field of view. This partial range is said to be about 50 mm in focal length. In an embodiment, the storage 204 holds a focal length value of 50 mm. If the human eye is regarded as a camera composed of a lens and an APS-C size image sensor, its focal length is about 35 mm. In this case, the storage 204 holds 35 mm as the focal length value.

The number of vertical pixels of the video is the number of vertical pixels of the video after being resized by the camera 10. For example, if the video size is resized to full HD (High Definition) size, the number of vertical pixels is 1080 pixels.

Also, the storage 204 holds spectator data 2043 . The audience data 2043 includes video data representing the audience. The image data representing the audience is, for example, the image data of the avatar of the audience. The video data representing the audience may be video data of the actual audience captured in advance. Furthermore, the spectator data 2043 may include biometric information such as heartbeats, motion states, and emotions of spectators watching the event at the venue as metadata. Such metadata is sequentially collected at the venue during the event and transmitted to the video processing device 20 .

The input device 205 is an interface device for the administrator of the video processing device 20 to operate the video processing device 20 . The input device 205 can include, for example, a touch panel, keyboard, mouse, various operation buttons, various operation switches, and the like. Input device 205 may be used to input variables 2042, for example.

The communication module 206 is a module that includes circuits used for communication between the video processing device 20 and other devices. The communication module 206 may be, for example, a communication module conforming to the wired LAN standard. Also, the communication module 206 may be a communication module conforming to the wireless LAN standard, for example.

FIG. 3 is a functional block diagram of the video processing device 20. As shown in FIG. As shown in FIG. 3 , the image processing device 20 has a receiving section 2011 , a distance estimating section 2012 , an image processing section 2013 , a three-dimensional processing section 2014 and a transmitting section 2015 . By executing the program 2041, the processor 201 of the video processing device 20 can operate as a reception unit 2011, a distance estimation unit 2012, an image processing unit 2013, a three-dimensional processing unit 2014, and a transmission unit 2015. The receiving unit 2011 , the distance estimating unit 2012 , the image processing unit 2013 , the three-dimensional processing unit 2014 and the transmitting unit 2015 may be realized by hardware different from the processor 201 .

The receiving unit 2011 acquires the video received from the camera 10 via the communication module 206, and decomposes the acquired video into frames. Then, the receiving unit 2011 sequentially transfers the images in units of frames to the distance estimation unit 2012 and the image processing unit 2013 . For example, when the video frame rate is 60 fps (frame per second), the receiving unit 2011 decomposes the video into 60 frames.

The distance estimating unit 2012 uses the image transferred from the receiving unit 2011 to determine the distance between the user U and the performer P when it is assumed that the user U was watching the scenery in the same range as the image captured by the camera 10 at the venue. Estimate virtual viewing distance. The operation of distance estimation section 2012 will be described later in detail.

The image processing unit 2013 processes the image captured by the camera 10 based on the viewing distance estimated by the distance estimation unit 2012 and the height of the performer P. For example, the image processing unit 2013 enlarges or reduces the area of the performer P in the image so that the height of the performer P in the three-dimensional space perceived by the user U by the three-dimensional display matches the actual height of the performer P. . The operation of the image processing unit 2013 will be described later in detail.

The 3D processing unit 2014 performs rendering processing for 3D display on the display 30 . For example, the 3D processor 2014 places the image plane of the performer P at the viewing distance estimated by the distance estimator 2012 in the 3D space perceived by the user U. Also, the three-dimensional processing unit 2014 arranges the image representing the audience in front of the image plane of the performer P based on the viewing distance estimated by the distance estimation unit 2012 . Then, the 3D processing unit 2014 renders the 3D video data in consideration of the reflection from the virtual light source in the 3D space. Then, the 3D processing unit 2014 captures a 3D space including the image plane of the performer P and the image plane of the audience from the position of the user U with a virtual stereo camera, and obtains a right eye image and a left eye image. and get. Then, the 3D processing unit 2014 transfers 3D image data including the acquired right-eye image and left-eye image to the transmission unit 2015 . The operation of the 3D processing unit 2014 will be described later in detail.

The transmission unit 2015 transmits the 3D video data sent from the 3D processing unit 2014 to the display 30 via the communication module 206 .

Next, the operation of the video distribution system 1 according to the embodiment will be described. FIG. 4 is a flow chart showing the operation of the video processing device 20. As shown in FIG. The processing of FIG. 4 is performed by the processor 201 at regular intervals, for example, from the start of an event until the end of the event. This certain period of time is, for example, a time interval at which image data is transmitted from the camera 10 .

In step S1, the processor 201 acquires the variable 2042 from the storage 204. As described above, variables 2042 include the length of the reference object, the focal length, the vertical size of the image sensor, and the number of vertical pixels of the image. In the example below, the reference length is the performer P's height.

In step S2, the processor 201 acquires image data transmitted from the camera 10 and stored in the RAM 203, for example. The processor 201 then decomposes the video into frames. The processor 201 also acquires spectator data 2043 from the storage 204 . In the following example, the video representing the spectator in the spectator data 2043 is the spectator's avatar video.

In step S3, the processor 201 performs distance estimation processing. The distance estimation process will be described below. Here, for the following explanation, it is assumed that the value of the variable 2042 is as follows.
Performer P's height Hp = 1.7 (m)
Focal length f=50 (mm)
Vertical size of image sensor S = 24 (mm)
Number of vertical pixels of video h = 1080 (pixels)

In the distance estimation process, the processor 201 first detects an object in each frame unit video. Object detection in the embodiment is a process of detecting an object with a known length in an image. In an embodiment, processor 201 detects performer P. Object detection techniques may be based on object detection algorithms such as Mask R-CNN and YOLO. In the following, it is assumed that the number of vertical pixels hp of the region of the performer P occupying the video is estimated to be 640 (pixels) by object detection.

After the object detection, the processor 201 estimates the height direction imaging range H(m) of the camera 10 at the performer P's standing position based on the performer's P height Hp. FIG. 5 is a diagram showing the relationship between the height Hp of the performer P and the shooting range H. As shown in FIG. The ratio between the height Hp of the performer P and the shooting range H shown in FIG. 5 matches the ratio between the vertical pixel number hp of the performer P and the vertical pixel number h of the video. Therefore, the following relationship (Equation 1) holds.
Hp/H=hp/h (Formula 1)
Therefore, H=h/hp×Hp=2.87 (m).

The photographing range H is the photographing range when photographed by the camera 10 having a certain focal length arranged at a certain distance from the performer P. FIG. Therefore, as shown in FIG. 6, the viewing distance D when it is assumed that the user U was looking at the scenery in the same range as the shooting range H is the viewing distance D of the user U's eyes with the lens of the focal length f and the vertical size S is calculated from the following (Equation 2).
S/f=H/D (Formula 2)
Therefore, D=H*f/S=5.98(m). This completes the distance estimation processing. After that, the processor 201 moves the process to step S4.

Now, return to the description of FIG. In step S4, the processor 201 processes the video. That is, the processor 201 enlarges or reduces the image so that the height of the performer P on the virtual display surface perceived by the user U by the three-dimensional display becomes Hp, which is the actual height.

In step S5, the processor 201 performs 3D video generation processing. Specifically, the processor 201 places the image plane of the performer P at a viewing distance D from the user U. In addition, the processor 201 places the image plane of the audience avatar between the user U and performer P image planes. Then, the processor 201 acquires a right-eye image and a left-eye image obtained when the three-dimensional space is photographed from the position of the user U with a virtual stereo camera. Here, the viewing distance D may determine the number and size of the audience avatars in the avatar video superimposed on the performer P video. Specifically, the number of audience avatars decreases as the viewing distance D decreases, and becomes 0 when the viewing distance D is equal to or less than the threshold. This is to reproduce the fact that the closer the audience seats are to the stage S, the smaller the number of spectators sitting in front of them in an event at a real venue. Therefore, the threshold of the viewing distance D can be set to the distance between the front rows of audience seats and the stage S in the venue, for example. Also, the size of the spectator avatar becomes larger at a position closer to the user U in the three-dimensional space perceived by the user through three-dimensional display, and becomes smaller at a position farther away. This is to reproduce the fact that in an event at a real venue, the closer the audience, the larger the audience. Further, in step S5, the processor 201 calculates the appearance of the performer P and the avatar at the viewing distance D in consideration of the reflection and scattering by the virtual light source that illuminates the performer P and the avatar, respectively, using an existing gloss reproduction technology. do. Furthermore, the processor 201 may perform processing such as moving the avatar according to the biological data of the audience.

In step S<b>6 , the processor 201 transmits the 3D image data including the right eye image and the left eye image to the display 30 via the communication module 206 . The display 30 appropriately displays the right-eye image and the left-eye image on the binocular display units based on the received 3D image data. Thereby, the user U perceives the image of the performer P whose height is Hp at the position of the viewing distance D.

In step S7, the processor 201 determines whether or not to end the process. For example, when the event ends and the communication with the camera 10 is cut off, it is determined that the processing ends. If it is not determined in step S7 that the process should end, the process returns to step S2. When it is determined in step S7 that the process should end, the processor 201 ends the process of FIG.

As described above, according to the embodiment, the imaging range of the camera 10 at the standing position of the performer P is estimated from the image captured by the camera 10 placed at the venue and the height of the performer P. Then, based on the estimated imaging range of the camera 10, the viewing distance D is estimated when the user U sees a scene in a range equivalent to the imaging range of the camera 10. FIG. Then, the virtual display plane is arranged so that the performer P with the height Hp is positioned at the viewing distance D in the three-dimensional space perceived by the user U by the three-dimensional display, and the size of the performer P on the virtual display plane is adjusted. . The three-dimensional image displayed in this way allows the user U to feel as if he were watching the performer P at the venue.

Furthermore, by superimposing the avatar video of the audience on the video of the performer P according to the viewing distance D, the user U can get the feeling of actually being at the venue.

[Modification 1]
Modifications of the embodiment will be described below. In the embodiments described above, the display 30 is assumed to be a non-transmissive wearable display. However, the display 30 may be a transmissive wearable display. As the transmissive wearable display, either a video transmissive wearable display or an optical transmissive wearable display may be used. A virtual image projection method and a retinal projection method are known as methods of projecting an image to a user U using a transmissive wearable display. In the embodiment, either a virtual image projection type or a retinal projection type wearable display may be used.

7A and 7B are diagrams showing a schematic configuration of a video distribution system 1 using a transmissive wearable display. Here, FIGS. 7A and 7B show only the remote configuration. The configuration is similar to that shown in FIG. 1, except for the configuration at the remote location.

FIG. 7A shows a first example. In the first example, the user U wears the display 30a, which is a transmissive wearable display, on the head. And there is a wall W in front of the user U. In the case of a transmissive wearable display, the user U perceives a real image obtained from the outside world in addition to the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience based on the display by the binocular display unit. Therefore, for example, if the viewing distance D from the user U is farther than the wall W, the user U perceives the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience through the wall. In other words, the user U gets a feeling as if the wall W is transparent and the space extends behind the wall W as well. By perceiving the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience through the wall in this way, the user U perceives the virtual image Pi of the performer P and the virtual image Ai of the audience avatar at an empty position in the air. You can get a sense of reality rather than doing it.

In the first example, the user U perceives the virtual image Pi of the performer P and the virtual image Ai of the audience's avatar behind the wall W. In this case, if the wall W has a clear unevenness, pattern, etc., the user U perceives the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience along with the unevenness, pattern, etc. of the wall, which impairs the sense of reality. . For this reason, it is desirable that the wall W be flat and plain. For similar reasons, it is desirable that the wall W be free of unnecessary objects.

FIG. 7B shows a second example. In a second example, the user U wears a display 30a, which is a transmissive wearable display, on the head. In front of the user U, there is a monitor M whose power is turned off. Therefore, if the position of the viewing distance D from the user U is farther than the screen of the monitor M, the user U perceives the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience through the screen of the monitor M. . In other words, the user U feels as if the space extends beyond the frame of the monitor M. By perceiving the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience through the monitor in this way, the user U perceives the virtual image Pi of the performer P and the virtual image Ai of the audience avatar at an empty position in the air. You can get a sense of reality rather than doing it. Further, in the second example, the user U can feel as if the image is being displayed on the monitor M even though the monitor M is turned off.

In the second example, the monitor M is powered off. This is because when an image is displayed on the monitor M, the user perceives the image displayed on the monitor M together with the virtual image Pi of the performer P and the virtual image Ai of the avatar of the audience. On the other hand, information related to the venue may be displayed on the monitor M.

[Modification 2]
In the above-described embodiment, processing is performed so that the height of the performer P is reproduced within the three-dimensional space. On the other hand, processing may be performed so that the size of the large service monitor 50 is reproduced within the three-dimensional space. In this case, processor 201 acquires an image from camera 40 . Here, the size of the virtual display surface is determined based on the height of the performer P in the video, as in the above-described embodiment. The size of the virtual display surface is determined so that the size of the surface matches the size of the large service monitor 50 . Furthermore, in Modified Example 2, the processor 201 arranges the image plane of the large service monitor 50 at a viewing distance position sufficiently far from the user U, and the image plane of the large service monitor 50 between the user U and the image plane of the large service monitor 50. An image representing the audience is placed at the placement position. As a result, a state is created in which only the distance from the user U is increased, and there are spectators between the image plane and the user, while maintaining the power and viewability of the image.

[Modification 3]
In the above-described embodiment, the image processing device 20 performs processing on images captured in real time by the camera 10 . On the other hand, processing by the video processing device 20 may be performed when playing back video recorded on a recording medium in the past. The video recorded on the recording medium is not necessarily limited to the video captured by the camera 10 . For example, the video recorded on the recording medium may be CG or the like. In the case of modification 3, the video recorded on the recording medium is once sent to the video processing device 20 . Then, the processor 201 of the video processing device 20 performs the processing shown in FIG. 4 and then transmits the 3D video data to the display 30 . It should be noted that the biological data of the spectator as the spectator data 2043 is data collected at the time of video recording. The processor 201 updates the avatar video using biometric data in synchronization with the timing of the video being reproduced.

[Modification 4]
In the embodiments described above, the display 30 is assumed to be a display capable of three-dimensional display. On the other hand, the display 30 may be a display incapable of three-dimensional display. In this case, the processor 201 does not have to change the size of the performer P in processing the video. Note that the processor 201 may enlarge or reduce the size of the performer P in the video so that the height of the performer P is reproduced if the screen of the display 30 is sufficiently large. On the other hand, the processor 201 scales up or down the size of the avatar to the size that would be perceived at a viewing distance D from the user U.

[Modification 5]
The above-described embodiments show applications in events such as music and theater. In contrast, embodiments may also be applied to events such as watching sports. In this case, the reference object does not necessarily have to be a person. For example, in an event such as a soccer match, the reference object may be a person such as a player, or an object such as a soccer ball. If the reference object is an object such as a soccer ball, the image may be enlarged or reduced with reference to the object as the reference object. Also, depending on the reference object, the horizontal length of the reference object may be used instead of the vertical length.

[Modification 6]
In the above-described embodiment, the imaging range of the camera 10 is estimated based on the length of the reference object. This estimation is particularly effective when the camera 10 is configured to be mobile and the imaging range can change over time. On the other hand, when the camera 10 is fixed, the shooting range of the camera 10 may be estimated from information such as the focal length and shooting distance of the camera 10 .

[Modification 7]
In the above-described embodiment, the sense of distance and the sense of size when it is assumed that the user U sees scenery in the same range as the shooting range of the camera 10 is reproduced in the three-dimensional image perceived by the user. On the other hand, the viewing distance D is defined as the distance from a specific audience seat to the performer P, and the image of the shooting range H equivalent to that seen by the user U from this viewing distance D is captured by the camera 10. A virtual display plane may be arranged so that a performer P with height Hp is positioned at a viewing distance D based on the trimmed video, and the size of the performer P on the virtual display plane may be adjusted. In this case, it is desirable that the camera 10 shoots a wide range with high resolution so that the image can be trimmed in various areas. This allows the user U to feel as if he were watching the performer P from a specific seat in the hall.

It should be noted that the present invention is not limited to the above-described embodiments, and can be variously modified in the implementation stage without departing from the gist of the present invention. Further, each embodiment may be implemented in combination as appropriate, in which case the combined effect can be obtained. Furthermore, various inventions are included in the above embodiments, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, if the problem can be solved and effects can be obtained, the configuration with the constituent elements deleted can be extracted as an invention.

DESCRIPTION OF SYMBOLS 1... Video distribution system 10... Camera 20... Video processing apparatus 30... Display 40... Camera 50... Large service monitor 201... Processor 202... ROM
203 RAM
204... Storage 205... Input device 206... Communication module 2011... Reception unit 2012... Distance estimation unit 2013... Image processing unit 2014... Three-dimensional processing unit 2015... Transmission unit 2041... Program 2042... Variable 2043... Spectator data

Claims

a receiving unit that receives video;
a distance estimating unit for estimating a viewing distance when it is assumed that a person is viewing a scene in the same range as the image;
an image processing unit that processes the image so that the size of the reference object appearing in the image at the viewing distance matches the actual size of the reference object;
a transmission unit that transmits the processed video to a display;
A video processing device comprising:
further comprising a three-dimensional processing unit that processes the video such that a virtual display plane on which a user perceives the video in a three-dimensional space is positioned at the viewing distance from the user;
The image processing unit enlarges or reduces the image so that the size of the reference object on the virtual display plane matches the actual size.
The video processing device according to claim 1.
The three-dimensional processing unit processes the video so that the video representing the audience is placed at a position shorter than the viewing distance from the user.
3. The video processing device according to claim 2.
The distance estimation unit
estimating the actual size of the range in which the image is displayed from the size of the reference object in the image;
estimating the viewing distance based on the actual size of the range in which the image is displayed, and the focal length of the lens and the size of the image sensor when the human is regarded as a camera having a lens and an image sensor;
The video processing device according to any one of claims 1 to 3.
the reference object is a human being,
wherein the size of the reference object is human height;
The video processing device according to any one of claims 1 to 4.
A video processing method executed by a video processing device,
Receiving video by a receiving unit of the video processing device;
estimating, by a distance estimation unit of the video processing device, a viewing distance when it is assumed that a human being has viewed scenery in a range equivalent to that of the video;
processing the image by the image processing unit of the image processing device so that the size of the reference object appearing in the image at the viewing distance matches the actual size of the reference object;
transmitting the processed video to a display by a transmission unit of the video processing device;
A video processing method comprising:
A video processing program for causing a computer to function as the receiving unit, the distance estimating unit, the video processing unit, and the transmitting unit of the video processing device according to any one of claims 1 to 5.