JP2022182120A - VIDEO GENERATION DEVICE, VIDEO PROCESSING SYSTEM, CONTROL METHOD AND PROGRAM FOR VIDEO GENERATION DEVICE - Google Patents

Abstract

A video generation device, a control method thereof, a video processing system, and a program are provided for shortening the time required to generate a virtual viewpoint video. Kind Code: A1 A video generation device for generating a virtual viewpoint video based on video data received from a plurality of video processing devices for processing video captured by a plurality of imaging devices, each of the plurality of video processing devices. a video receiving unit for receiving first video data on which predetermined processing has been performed; an effective subject determining unit that determines each time, and based on the determination result of the effective subject determining unit, the second video data that has not been subjected to the predetermined processing during the time when the subject exists, is transferred to each of the plurality of video processing devices. and a virtual viewpoint video generation unit that generates a virtual viewpoint video based on the second video data. [Selection drawing] Fig. 1

Description

The present invention relates to technology for generating a virtual viewpoint image.

In recent years, multiple image capturing devices have been installed at different positions, and from the synchronously captured multi-viewpoint images, a video processing device performs processes such as foreground/background separation, 3D shape estimation, 3D shape generation, and rendering. Techniques for synthesizing images that look like they are being photographed have been realized.

By applying this technology to the shooting of sports such as soccer and basketball, it has become possible for viewers to appreciate images taken by a cameraman from within the field during a match, which was not possible in the past. It is possible to create an image that gives a sense of presence. In addition, by applying it to the studio and shooting music videos, etc., it is possible to use any camera work, such as a bird's-eye view of the performer from above, which was difficult to shoot in the past, or a see-through image of the floor from below. can be photographed. Furthermore, since it is possible to selectively display or hide the photographed subject and the effective subject, it is possible to generate a video with a higher degree of freedom than conventional photographing by a cameraman.

Patent Literature 1 describes an apparatus that generates a virtual viewpoint image using image data transmitted from a plurality of sensor systems.

JP 2017-211828 A

However, with the technique described in Patent Document 1, for example, due to problems such as a large amount of image data, it takes time to transmit image data, and as a result, there is a problem that it takes time to generate a virtual viewpoint image. .

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for shortening the time required to generate a virtual viewpoint video.

A video generation device according to the present invention for achieving the above object includes:
A video generation device that generates a virtual viewpoint video based on video data received from a plurality of video processing devices that process video captured by a plurality of imaging devices,
receiving means for receiving first video data on which predetermined processing has been performed from each of the plurality of video processing devices;
Determination means for determining whether or not a subject exists within the range of the imaging angle of view of each imaging device based on the first video data, for each time;
requesting means for requesting, from each of the plurality of video processing devices, second video data for which the predetermined processing has not been performed during the time when the subject exists, based on the determination result of the determining means;
generating means for generating a virtual viewpoint image based on the second image data;
characterized by comprising

According to the present invention, it is possible to shorten the time required to generate a virtual viewpoint video.

1A shows a configuration example of a video processing system according to an embodiment; FIG. 1B shows a functional configuration example of a video processing device in the video processing system according to this embodiment; FIG. The figure explaining the process which concerns on one embodiment. 1 is a layout diagram of an imaging device according to an embodiment; FIG. 1 is a diagram showing an example of the hardware configuration of a video processing device and a video generation device according to one embodiment; FIG. FIG. 2 is a diagram showing an example of a functional configuration of a video generation device according to the first embodiment; FIG. 4 is a schematic diagram of a bounding box according to the first embodiment; FIG. FIG. 2 is a layout diagram of an imaging device and an imaging angle of view according to the first embodiment; 4A and 4B are diagrams related to determination of an effective subject according to the first embodiment; FIG. FIG. 4 is a view showing an effective subject list according to the first embodiment; FIG. 4 is a flowchart showing the procedure of processing performed by the video generation device according to the first embodiment; The figure which shows an example of the functional structure of the image|video production|generation apparatus which concerns on 2nd Embodiment. 10 is a flowchart showing the procedure of processing performed by the video generation device according to the second embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
In the present embodiment, in order to generate a high-quality virtual viewpoint image in a short time, image data corresponding to the time when the subject does not exist is not acquired, and image data corresponding to the time when the subject exists is acquired. An example of generating a virtual viewpoint video based on this will be described.

In this embodiment, virtual viewpoint images are generated using two types of systems. Problems in this case will be described first. As one of the virtual viewpoint video generation systems, there is a system that prioritizes video output time and is capable of high-speed processing (hereinafter, this system is referred to as a high-speed system). A high-speed system assigns a small-scale video processing device to each camera, distributes some of the video processing required to generate a virtual viewpoint video, and processes the pre-processed images in each video processing device. Speeding up is realized by concentrating on the server. However, due to the processing time constraint of generating a virtual viewpoint video in real time, it is difficult to process a large load and to improve the image quality.

On the other hand, as a system to solve the problem of high-speed systems, there is a system that generates a virtual viewpoint video by recording video taken by a camera on a recording device, etc., consolidating it, and processing it on a processing server ( Hereinafter, this system will be referred to as a high image quality system). In the high image quality system, it is possible to temporarily store the camera images captured by each image capturing device, so it is possible to generate a virtual viewpoint video over time after capturing. Therefore, it is possible to perform image processing with a large load using machine learning or the like, and it is possible to realize high image quality of the virtual viewpoint image. However, although the high image quality system does not require real-time performance unlike the high-speed system, it takes a long time to generate an image due to the image transmission time and the high-load image generation processing time.

In this embodiment, an example will be described in which data generated in a high-speed system is used to shorten the time required for image generation in a high-definition system.

<Configuration>
FIG. 1A is a diagram showing a configuration example of a video processing system according to this embodiment. The imaging devices 100a to 100z are connected to corresponding video processing devices 200a to 200z, respectively, and the signals output from them are aggregated in the video generating device 1000. FIG. Note that the number of photographing devices and image processing devices can be appropriately changed depending on the size of the photographing area and the photographing object.

FIG. 1B is a diagram showing a functional configuration example of a video processing device in the video processing system according to this embodiment. The photographing apparatus 100 includes a lens (not shown), an image sensor such as a CCD/CMOS, an image processing circuit, and the like. It transmits to the video processing device 200 .

The video processing device 200 includes a control unit 201 , a video processing unit 202 , a recording control unit 203 and a recording unit 204 .

Video data output from the imaging device 100 is delivered to the video processing unit 202 and the recording control unit 203 via the control unit 201 .

When the image data is transmitted to the image processing unit 202, the image processing unit 202 converts the image data into a foreground image (because it is a foreground image, the amount of data is smaller than that of the image data) obtained by cutting out the area of the effective subject. Here, the effective subject refers to an object extracted by subtracting a background image prepared in advance from the photographed image using the background subtraction method described later.

Furthermore, the video processing unit 202 transmits the converted foreground image to the video generation device 1000 as first video data. An example of background subtraction processing will now be described with reference to FIG. Here, it is assumed that an image 7100 is shot, and a background image 7010 is prepared in advance, which has the same angle of view as the image 7100 and is an image of a stadium, a wall surface of a studio, etc. for which virtual viewpoint video generation is not performed. By subtracting the background image 7010 from the image 7100, the effective subject can be extracted. An image 7200 is an image after subtraction, and an effective subject 7000 is extracted. Furthermore, an image 7300 obtained by cutting out an area including the effective subject from the image 7200 is output as a foreground image and transmitted to the image generation device 1000 .

On the other hand, when the video data is transmitted to the recording control unit 203, the recording control unit 203 temporarily holds the video data as second video data in an auxiliary storage device such as a volatile memory (not shown), and records it. Stored in unit 204 . Further, when a request is received from the image generating apparatus 1000 by a method to be described later, the recording control unit 203 selectively reads out the requested image data from the image data recorded in the recording unit 204 and generates an image. Send to device 1000 .

The recording unit 204 is composed of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD), and records and holds video data under the control of the recording control unit 203 . Furthermore, at the time of reading, the recording unit 204 reads predetermined video data out of the recorded video data and transmits it to the recording recording unit 203 under the control of the recording control unit 203 . Also, under the control of the recording control unit 203, recorded video data can be erased or overwritten.

Next, FIG. 3 shows an arrangement example of each imaging device from the imaging device 100a to the imaging device 100z. Each imaging device is arranged so as to surround an imaging area 3000, and a virtual viewpoint image is generated for an object inside the imaging area 3000 surrounded by points O, X, Y, and Z.

<Hardware configuration of video generation device and video processing device>
FIG. 4 is a diagram showing an example of the hardware configuration of the image generation device 1000 according to this embodiment. Although the video generation device 1000 will be described below as an example, the hardware configuration of the video processing device 200 is the same. The image generation device 1000 includes a CPU 1001, a main memory device 1002, an auxiliary memory device 1003, an external I/F 1004, and a GPU 1005, which are connected via an internal bus 1006 so as to be able to communicate with each other.

A CPU 1001 is a central processing unit, and controls the image generating apparatus 1000 such as various calculations, image processing, and data input/output.

The main storage device 1002 is a storage device that functions as a work area for the CPU 1001 and a temporary storage area for data. The main storage device 1002 is implemented using storage media such as Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM).

The auxiliary storage device 1003 is a storage device that stores information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape data, and two-dimensional maps. The auxiliary storage device 1003 is implemented using storage media such as read only memory (ROM), nonvolatile memory such as flash memory, hard disk drive (HDD), solid state drive (SSD), and tape media. A plurality of them may be combined in order to increase capacity and speed, or may be logically integrated into one such as RAID (Redundant Arrays of Inexpensive Disks).

The external I/F 1004 is a communication interface used for communication with external devices such as cameras, PCs, and devices, an interface used for connection with external devices such as joysticks, keyboards, and mice that accept various user operations. A general-purpose input/output interface or the like for connecting an external storage and performing file input/output. The external I/F 1004 is composed of an interface having a physical cable connection terminal such as InfiniBand, Ethernet, Universal Serial Bus (USB), or a wireless interface such as wireless LAN or Bluetooth. In the high-speed system, video data output from the video processing unit 202 is aggregated by a switching hub (not shown) or the like and input to the external I/F 1004 .

The GPU 1005 is an arithmetic unit for executing image processing operations at high speed, and has a function of rendering an image from a virtual viewpoint and generating shape data and color information necessary for a virtual viewpoint video. The GPU 1005 may include a separate main memory device different from the main memory device 1002 . Also, the GPU 1005 may perform part or all of the various operations performed by the CPU 1001 other than those described above.

The bus 1006 is configured by a PCI Express bus or the like, and each part of the hardware block can perform two-way communication. Any bus 1006 may be used as long as each part of the hardware block can communicate bidirectionally. Note that the configuration shown in FIG. 4 is an example, and shall be appropriately changed and configured according to the object to be photographed and the function.

<Functional Configuration of Image Generation Device>
FIG. 5 is a diagram showing the functional configuration of the video generation device 1000 according to this embodiment. The video generation device 1000 includes a video reception unit 1011, a three-dimensional shape generation unit 1012, a subject position detection unit 1013, an effective subject determination unit 1014, a video selection reception unit 1015, a foreground extraction unit 1016, a three-dimensional shape generation unit 1017, and a virtual object. A viewpoint video generation unit 1018 is provided. The function of each processing unit of the image generation device 1000 is realized by the CPU 1001 developing a computer program stored in the auxiliary storage device 1003 into the main storage device 1002 and executing it.

The video reception unit 1011 receives the foreground image 7300 processed by each video processing device 200 and transmits it to the three-dimensional shape generation unit 1012 .

Based on the foreground image 7300 transmitted from the video receiving unit 1011, the 3D shape generation unit 1012 generates a 3D shape of the effective object by a 3D shape restoration method such as the visual volume intersection method or Multi-View-Stereo (MVS). to generate

A subject position detection unit 1013 defines a rectangular parallelepiped (hereinafter referred to as a bounding box) that circumscribes the three-dimensional shape generated by the three-dimensional shape generation unit 1012 . FIG. 6 shows the relationship between the subject and the bounding box. Here, a rectangular parallelepiped with points a, b, c, d, e, f, g, and h as vertices is defined as the bounding box 5100 so as to circumscribe the three-dimensional shape of the subject 5000. . Furthermore, the information of the three-dimensional coordinates of this bounding box is stored in the main storage device 1002 of the image generation device 1000 .

A valid subject determination unit 1014 determines whether or not a valid subject exists in the imaging range of each imaging device 100 . Here, in the auxiliary storage device 1003 of the image generation device 1000, data of the three-dimensional coordinates of the photographing range photographed by each of the photographing devices 100a to 100z is held in advance. FIG. 7 shows an example of three-dimensional coordinate data of the imaging range. Here, an arbitrary photographing device 100n among the photographing devices will be described. A three-dimensional image having points o, p, q, r, s, t, u, and v as vertices is set as an imaging range 4000n for the imaging device 100n. coordinates are kept. Furthermore, it is assumed that a photographing range corresponding to the arrangement and photographing angle of view is set for each photographing device from the photographing device 100a to the photographing device 100z. It should be noted that the arrangement of each photographing device and the size and shape of the photographing range shown in FIG. shall be set to

A valid subject determination unit 1014 determines whether or not a valid subject exists in the imaging range of each imaging device, using the three-dimensional coordinate data of the imaging range shot by each imaging device.

Here, with reference to FIG. 8, a procedure for determining whether or not a subject exists in the photographing range 4000n and photographing range 4000m of arbitrary photographing devices 100n and 100m among the photographing devices will be described. As shown in FIG. 8, it is assumed that a subject 5000 exists, a bounding box 5100 is defined for this subject, and three-dimensional coordinates of this bounding box are calculated by the subject position detection unit 1013 .

Here, if there is at least one bounding box partially or wholly included in the imaging range of the imaging device, it is determined that an effective subject exists in the imaging range of the imaging device.

For example, in FIG. 8, since the bounding box 5100 exists within the imaging range 4000n, it is determined that an effective subject exists for the imaging device 100n, and the determination result "valid subject exists" is entered in the effective subject determination result list 6000 described later. " is output. On the other hand, since no bounding box exists within the imaging range of 4000 m, it is determined that there is no valid subject for the imaging apparatus 100 m, and the determination result "no effective subject" is output to the effective subject determination result list 6000 described later. .

Blank data for the effective subject determination result list 6000 as shown in FIG. 9 is prepared in advance and stored in the auxiliary storage device 1003 . Then, the determination result for each imaging device is updated, and the determination result at a certain time is stored in the effective subject determination result list 6000 . Also, the effective subject determination result list 6000 is updated at time intervals that are appropriately set according to the frame rate of the imaging device and the state of the subject. For example, the update time interval is shortened for a fast-moving subject, and the update time interval is lengthened for a slow-moving subject. Note that the specifications of this list are only an example, and the data format can be changed to another format as long as at least information about whether or not there is an effective subject in each imaging device at the time of shooting is included. may be converted into data by adding information such as

Based on the determination result of the effective subject determination unit 1014, the video selection reception unit 1015 requests and receives the video data of the imaging device in which the effective subject exists from the recording control unit 203 of the video processing device corresponding to the imaging device. do. Specifically, based on the effective subject determination result list 6000, the image data of the time when the effective subject exists in each photographing device is requested and received. Video selection reception unit 1015 transmits the received video data to foreground extraction unit 1016 .

Foreground extraction section 1016 extracts a foreground image from the video data output from video selection reception section 1015 . In the processing here, a foreground image is generated by applying a machine learning method to video data, using a CNN (Convolutional neural network) or the like, and extracting an effective subject by comparing with a database prepared in advance. As a result, it is possible to extract a foreground image with high accuracy even for video data such as a densely populated subject, for which it is difficult to generate an accurate foreground image by the background subtraction method.

A three-dimensional shape generation unit 1017 generates a three-dimensional shape for the foreground image output from the foreground extraction unit 1016 by a three-dimensional shape restoration technique. Here, since the high-precision foreground image output from the foreground extraction unit 1016 is used, it is possible to generate a three-dimensional shape that is closer to the actual object shape than the three-dimensional shape generated by the three-dimensional shape generation unit 1012. can.

The virtual viewpoint video generation unit 1018 displays how the 3D shape generated by the 3D shape generation unit 1017 from the virtual viewpoint is displayed based on information about the position and direction of the virtual viewpoint input by an input unit (not shown). 3D calculation of whether Then, the images are colored using the image data of the corresponding photographing device to generate a virtual viewpoint image.

<Processing>
Next, a procedure of processing performed by the image generation device 1000 according to this embodiment will be described with reference to the flowchart of FIG. 10 .

In S100, the video receiving unit 1011 receives foreground images output from the video processing devices 200a to 200z corresponding to the imaging devices 100a to 100z.

In S101, the 3D shape generation unit 1012 generates a 3D shape using a 3D shape restoration technique such as the visual volume intersection method or Multi-View-Stereo (MVS) based on the foreground image received in S100. do.

In S102, the subject position detection unit 1013 defines a bounding box for the three-dimensional shape generated in S101, and stores three-dimensional coordinate information of the bounding box as coordinate information of the subject in the main storage device 1002 of the video generation device 1000. Store.

In S103, the effective subject determination unit 1014 determines whether or not a valid subject exists in the imaging range of each imaging device using the determination method described above with reference to FIG. Here, determination is sequentially performed for the imaging devices 100a to 100z, and if it is determined that even one valid bounding box exists in the imaging range of the imaging device, the determination is terminated at that point, and the next Starts determination of the shooting range of . In S104, the effective subject determination unit 1014 outputs an effective subject determination result list based on the determination result of S103. Here, the list 6000 described above with reference to FIG. 9 is output.

In S105, based on the effective subject determination result list 6000 output in S104, the video selection reception unit 1015 transmits video data of the time when the effective subject exists for each imaging device via the external I/F 1104. Then, a request is sent to the image processing device corresponding to the corresponding image capturing device.

In S<b>106 , the video selection reception unit 1015 receives the video data requested in S<b>105 and holds it in the auxiliary storage device 1103 . In S107, the foreground extraction unit 1016 extracts a foreground image from the video data received in S106 using the above-described machine learning.

In S108, the three-dimensional shape generation unit 1017 generates a three-dimensional shape for the foreground image extracted in S107 by a three-dimensional shape restoration method.

In S109, the virtual viewpoint video generation unit 1018 determines how the three-dimensional shape generated in S108 is displayed from the virtual viewpoint based on the information on the position and direction of the virtual viewpoint input by the input unit (not shown). is calculated three-dimensionally, and is colored using the image data of the corresponding photographing device to generate a virtual viewpoint image. Thus, the series of processing in FIG. 10 ends.

As described above, in the present embodiment, image data corresponding to the time when the subject does not exist is not acquired, but image data corresponding to the time when the subject exists is acquired, and a virtual viewpoint image is generated based on the image data.

As a result, instead of sending video data from all camera devices, transmission time can be reduced by selecting the camera device and sending the video data. can be generated with

(Second embodiment)
In this embodiment, when video data is recorded in the recording unit of the video processing device, only the video data in which an effective subject exists is selectively recorded to reduce the video data amount, thereby reducing the transmission time. An example of realizing

<Configuration>
Since the configuration of each device and the hardware configuration of the video processing system according to the present embodiment are the same as the configuration described with reference to FIGS. 1 and 4 of the first embodiment, description thereof will be omitted.

FIG. 11 is a functional block diagram showing the functional configuration of the image generation device 1000 according to this embodiment. It is assumed that these functional blocks are realized by developing a computer program stored in the auxiliary storage device 1003 in the main storage device 1002 and executing it by the CPU 1001 in FIG.

The configuration of this embodiment is the same as that of the first embodiment, except that a recording selection unit 2011 connected to an effective subject determination unit 1014 is added, and a video selection reception unit 1015 is replaced with a video reception unit 2012. Functional blocks with the same reference numerals are the same as the functional blocks described in the first embodiment. Functions of the recording selection unit 2011 and the video reception unit 2012 will be described below.

The recording selection unit 2011 receives the effective subject determination result list output from the effective subject determination unit 1014, and transmits a signal for changing the recording state to the recording control unit 203 of the video processing device 200 based on the effective subject determination result list. do. Here, as a signal for changing the recording state, a signal requesting to hold the video data is transmitted to the recording control unit of the video processing device corresponding to the imaging device 100 determined to have an effective subject. In addition, a signal requesting stop holding of video data is transmitted to the recording control unit 203 of the video processing device 200 corresponding to the imaging device determined that no effective subject exists. Here, the signal may be continuously transmitted during the time interval during which the effective subject determination unit 1014 determines the effective subject, or the trigger signal may be transmitted only when the state changes. good.

In addition, the recording control unit 203 of the video processing device 200 that has received the signal records the video data received from the image capturing device 100 when receiving the signal to hold the video data, and receives the signal to stop recording the video data. is received, video data recording is stopped. In this process, the recording of the corresponding video data may be held or stopped. Alternatively, only the image data to be held may be left out of the image data recorded in the recording unit 204, and the other image data may be erased.

The image receiving unit 2012 receives all the image data from the image processing apparatuses 200a to 200z instead of selectively receiving image data like the image selection receiving unit 1015 described in the first embodiment. , to the foreground extraction unit 1016 . The image data received here is image data in which an effective subject exists.

<Processing>
Next, a procedure of processing performed by the video generation device 1000 according to this embodiment will be described with reference to the flowchart of FIG. 12 .

In S200, the image receiving unit 1011 receives a foreground image output from the image processing device 200a corresponding to the image capturing device 100a to the image processing device 200z.

In S201, the 3D shape generation unit 1012 generates a 3D shape by a 3D shape restoration technique such as the visual volume intersection method or Multi-View-Stereo (MVS) based on the foreground image received in S200. In S<b>202 , the subject position detection unit 1013 defines a bounding box for the three-dimensional shape generated in S<b>201 and stores the three-dimensional coordinate information of the bounding box in the main storage device 1002 of the video generation device 1000 .

In S<b>203 , the effective subject determination unit 1014 determines whether or not there is a valid subject within the imaging range of each imaging device 100 . It is determined using the determination method described with reference to FIG. In S204, the effective subject determination unit 1014 outputs an effective subject determination result list based on the determination result in S203. Here, the list 6000 described with reference to FIG. 9 is output.

In S205, based on the effective subject determination result list 6000 output in S204, the recording selection unit 2011 records the video data for the corresponding time period if there is an effective subject for each imaging device 100. A request is transmitted to the image processing apparatus 200 corresponding to the corresponding image capturing apparatus 100 via the external I/F 1104 . On the other hand, if the effective subject does not exist, a request to stop recording is transmitted to the image processing apparatus 200 corresponding to the corresponding image capturing apparatus 100 via the external I/F 1104 . As a result, only the video data of the imaging device in which a valid subject exists is recorded in the recording unit 204 of the corresponding video processing device 200 .

In S<b>206 , the video reception unit 2012 receives the video data recorded in response to the request in S<b>205 from the recording unit 204 of the video processing device 200 corresponding to each imaging device 100 and holds it in the auxiliary storage device 1103 .

In S207, the foreground extraction unit 1016 extracts a foreground image from the video data received in S206 using the above-described machine learning. In S208, the 3D shape generation unit 1017 generates a 3D shape for the foreground image extracted in S207 by a 3D shape restoration technique.

In S209, the virtual viewpoint video generation unit 1018 determines how the three-dimensional shape generated in S208 is displayed from the virtual viewpoint based on the information on the position and direction of the virtual viewpoint input by the input unit (not shown). is calculated three-dimensionally, and is colored using the image data of the corresponding photographing device to generate a virtual viewpoint image. Thus, the series of processing in FIG. 12 ends.

As described above, in the present embodiment, only video data in which a subject exists is recorded and held in the video processing device. Generate.

As a result, instead of transmitting image data from all image capturing devices in the past, the amount of image data can be reduced by selecting the image capturing device and recording the image data. A virtual viewpoint video can be generated in a short time.

In addition, you may combine a part of above-mentioned each embodiment suitably. Alternatively, the processing of the first embodiment may be applied according to whether or not a specific subject is present within the angle of view of the photographing device using a technique such as image recognition, or the processing of the second embodiment may be applied. process may be applied.

For example, the image processing apparatus 1000 is further provided with a recognition unit for recognizing a specific subject, and the subject determination unit 1014 determines in S103 of FIG. may be configured to be determined every time. This makes it possible to generate a virtual viewpoint video limited to a specific subject in a short time.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: photographing device, 200: video processing device, 1000: video generation device, 201: control unit, 202: video processing unit, 203: recording control unit, 204: recording unit, 1011: video reception unit, 1012: three-dimensional shape Generating unit 1013: subject position detecting unit Effective subject determining unit 1014, 1015: video selection receiving unit 1016: foreground extracting unit 1017: three-dimensional shape generating unit 1018: virtual viewpoint video generating unit

Claims (10)

A video generation device that generates a virtual viewpoint video based on video data received from a plurality of video processing devices that process video captured by a plurality of imaging devices,
receiving means for receiving first video data on which predetermined processing has been performed from each of the plurality of video processing devices;
Determination means for determining whether or not a subject exists within the range of the imaging angle of view of each imaging device based on the first video data, for each time;
requesting means for requesting, from each of the plurality of video processing devices, second video data for which the predetermined processing has not been performed during the time when the subject exists, based on the determination result of the determining means;
generating means for generating a virtual viewpoint image based on the second image data;
A video generation device comprising: 2. The video generating apparatus according to claim 1, wherein said receiving means receives, as said first video data, video data of a foreground image representing said subject. shape generating means for generating a three-dimensional shape of the subject based on the first image data of the foreground image;
further comprising acquisition means for acquiring coordinate information indicating the position of the subject based on the three-dimensional shape,
3. The video generation apparatus according to claim 2, wherein the determination means determines whether or not the subject exists within the range of the shooting angle of view of each shooting device based on the coordinate information. . further comprising recognition means for recognizing a specific subject,
2. The judging means judges whether or not the specific subject exists within the range of the photographing angle of view of each photographing device based on the first image data at each time. 4. The video generation device according to any one of items 1 to 3. A video image corresponding to the photographing device determined that the subject does not exist, based on the determination result of the determining means, a signal requesting to stop recording the second image data during the time when the subject does not exist. 5. A video production device according to any one of claims 1 to 4, further comprising transmitting means for transmitting to the processing device. a signal requesting recording of the second image data at the time when the subject exists, based on the determination result of the determining means, to the video processing device corresponding to the photographing device determined that the subject exists; 5. The image generating apparatus according to any one of claims 1 to 4, further comprising transmitting means for transmitting. 7. The image generating apparatus according to claim 1, wherein the data amount of said first image data is smaller than the data amount of said second image data. a plurality of image capturing devices; a plurality of image processing devices for processing images captured by the plurality of image capturing devices; A video processing system comprising a device,
each of the plurality of video processing devices,
image processing means for performing predetermined processing on video data received from a corresponding imaging device to generate first video data;
recording means for recording video data received from the corresponding imaging device as second video data,
The video generation device is
receiving means for receiving the first video data from each of the plurality of video processing devices;
Determination means for determining whether or not a subject exists within the range of the imaging angle of view of each imaging device based on the first video data, for each time;
requesting means for requesting each of the plurality of video processing devices for the second video data of the time when the subject exists, based on the determination result of the determining means;
generating means for generating a virtual viewpoint image based on the second image data received from each of the plurality of image processing devices and containing the subject;
A video processing system comprising: A control method for a video generation device that generates a virtual viewpoint video based on video data received from a plurality of video processing devices that process video captured by a plurality of imaging devices, comprising:
a receiving step of receiving first video data on which predetermined processing has been performed from each of the plurality of video processing devices;
a determination step of determining, based on the first video data, whether or not a subject exists within the range of the imaging angle of view of each imaging device at each time;
a requesting step of requesting, from each of the plurality of video processing devices, second video data for which the predetermined processing has not been performed during the time when the subject exists, based on the determination result of the determining step;
a generation step of generating a virtual viewpoint image based on the second image data;
A control method for a video generation device, comprising: A program for causing a computer to function as the image generation device according to any one of claims 1 to 7.

Images