JP2018191236A - Information processing system, information processing method, apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control transmission of an image of a subject according to a position of the subject.SOLUTION: Camera adapters 120a to 120z control transmission of data including images of a region of a subject included in a photographed image photographed by cameras 112a to 112z on the basis of a position of the subject in the real space.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing method, an apparatus, and a program.

  There is a technique in which a plurality of cameras installed at different positions perform shooting synchronously, and virtual viewpoint content is generated using images taken from a plurality of viewpoints obtained by the shooting. According to such a technique, for example, since a highlight scene of soccer or basketball can be viewed from various angles, it is possible to give the user a higher sense of realism than a normal image.

  By the way, in such a technique, a subject region may be cut out from a captured image and transmitted. As a technique for extracting and transmitting a necessary area from an image photographed by a camera, there is a technique described in Patent Document 1. Patent Document 1 discloses the following technique. That is, a plurality of surveillance cameras are installed so that an area including the vicinity of the entrance / exit of a train that is stopped at the station platform can be photographed from the outside of the train. Only images near the passenger door are cut out from the images taken by the plurality of monitoring cameras, transmitted, and collected.

JP 2014-192844 A

However, the technique described in Patent Document 1 transmits an image in the vicinity of a passenger door whose position does not change relatively. Therefore, the technique disclosed in Patent Document 1 does not consider transmission of an image of a subject when the position where the subject exists may change.
The present invention has been made in view of such problems, and an object thereof is to control transmission of an image of a subject in accordance with the position of the subject.

  An information processing system according to the present invention includes an acquisition unit that acquires an image of a subject area included in a captured image, and transmission of data including the image of the subject area included in the captured image based on the position of the subject. And a control means for controlling.

  According to the present invention, transmission of an image of a subject can be controlled according to the position of the subject.

It is a figure which shows the structure of an image processing system. It is a figure which shows the structure of a camera adapter. It is a figure which shows the structure of an image process part. It is a top view which shows all the common exposure ranges. It is a perspective view which shows all the common exposure ranges. It is a figure which takes out and shows the part corresponding to three cameras of the upstream from FIG. It is a figure which shows an example of a foreground image. It is a flowchart which shows the 1st example of a transmission adjustment process. It is a figure which shows the 1st example of the object transmitted to a subordinate camera. It is a flowchart which shows the 1st example of a transmission adjustment process. It is a figure which shows the 2nd example of the object transmitted to a subordinate camera.

Hereinafter, embodiments will be described in detail with reference to the drawings.
When generating and viewing virtual viewpoint content based on images taken from a plurality of viewpoints, first, images taken by a plurality of cameras are collected in an image processing unit such as a server. Next, the image processing unit generates a virtual viewpoint content by performing processing such as generation of a three-dimensional model and rendering using images taken by a plurality of cameras, and transmits the virtual viewpoint content to the user terminal. Stereo matching is a typical method for sequentially generating a three-dimensional model of a moving subject, such as a player, from a captured image.

  In order to perform stereo matching, the elements of each surface constituting the subject need to be photographed by at least two cameras. For this reason, in order to generate a 3D model with no omission of the surface of the subject, the subject is a common range of the subject range (field of view) that is the photographing range that can be taken by all cameras (hereinafter referred to as necessary). (Referred to as the “common common coverage area”). In other words, the image of the subject located outside the entire common subject range is data that does not contribute to the generation of a complete three-dimensional model.

  However, depending on the content of the virtual viewpoint content, there may be a case where a complete three-dimensional model is not necessary, for example, a three-dimensional model for only one half of the subject is required. However, even in such a case, an image of a subject that has been captured only from one camera cannot be used to generate a three-dimensional model. Transmitting image data that does not contribute to generation of the three-dimensional model to the image processing unit unnecessarily imposes a transmission load between the camera and the image processing unit.

If an attempt is made to reduce the transmission load while including images that do not contribute to the generation of the 3D model, relatively high compression will be performed including the images necessary for the generation of the 3D model. Degraded image quality.
Therefore, in the present embodiment, it is determined whether or not it is necessary to generate a three-dimensional model for each of the regions of the subject included in the captured images captured from a plurality of viewpoints, and transmission and image quality are determined according to the determination result. The case of controlling is described as an example. By doing so, it is possible to maintain high image quality necessary for generating the three-dimensional model while reducing the transmission load.

<First Embodiment>
First, the first embodiment will be described.
(Image processing system)
FIG. 1 is a diagram illustrating an example of the configuration of the image processing system 100. In this embodiment, a case where a plurality of cameras and a plurality of microphones are installed in a facility such as a stadium (stadium) or a concert hall will be described as an example. The image processing system 100 performs shooting and sound collection using a plurality of cameras and a plurality of microphones. The image processing system 100 includes sensor systems 110a to 110z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190. In the present embodiment, for example, an example of an information processing system is realized by the image processing system 100.

  The controller 300 includes a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management, parameter setting, and the like through the networks 310 a to 310 c, 180 a to 180 b, and 170 a to 170 y for each block constituting the image processing system 100. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE which is an Ethernet (registered trademark) standard, or may be configured by combining an interconnect (Infiniband, etc.), an industrial Ethernet, or the like. Further, the network is not limited to these, and may be another type of network.

First, an example of an operation for transmitting images and sounds obtained from 26 sets of the sensor systems 110a to 110z from the sensor system 110z to the image computing server 200 will be described. In the present embodiment, a case where the sensor systems 110a to 110z are connected to each other in a daisy chain will be described as an example.
In the present embodiment, when there is no special description, the 26 sets of the sensor systems 110a to 110z are not distinguished and are described as the sensor system 110. Similarly, the devices and networks in each sensor system 110 are not distinguished from each other unless there is a special description, and the microphone 111, the camera 112, the pan head 113, the external sensor 114, the camera adapter 120, and the network 170 are not distinguished. It describes.

  In the present embodiment, the number of sensor systems 110 is 26 sets, but this is an example, and the number of sensor systems 110 is not limited to 26 sets. In the present embodiment, unless otherwise specified, an image is described as including a concept of a moving image and a still image. That is, the image processing system 100 according to the present embodiment can process both still images and moving images. In the present embodiment, an example in which the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual viewpoint sound will be mainly described. However, the virtual viewpoint content is not limited to this. For example, audio may not be included in the virtual viewpoint content. Further, for example, the sound included in the virtual viewpoint content may be a sound collected by a microphone located closest to the virtual viewpoint. Further, in the present embodiment, for the sake of simplification of explanation, detailed explanation of the sound is partially omitted, but the sound is basically processed together with the image.

  Each of the sensor systems 110a to 110z includes one camera 112a to 112z. That is, the image processing system 100 includes a plurality of cameras for photographing the subject from a plurality of directions. The plurality of sensor systems 110 are connected to each other in a daisy chain. With this connection form, it is possible to reduce the number of cables used for connecting the plurality of sensor systems 110 and save labor in wiring work. In particular, this connection mode is effective when the resolution of captured images is increased to 4K, 8K, or the like, and the capacity of image data is increased due to an increase in frame rate.

  The connection form of the plurality of sensor systems 110 is not limited to the daisy chain. For example, a star type network may be used. When a star network is used, all the sensor systems 110a to 110z are connected to the switching hub 180. In this case, data transmission / reception is performed between the sensor systems 110a to 110z via the switching hub 180.

  Further, FIG. 1 shows a configuration in which all of the sensor systems 110a to 110z are cascade-connected so as to be daisy chain connected. However, not all of the sensor systems 110a to 110z need be cascaded. For example, the plurality of sensor systems 110 may be divided into a plurality of groups. In this case, the sensor system 110 may be connected so as to be daisy chain connected in divided group units. Alternatively, the camera adapter 120 that is the end of the divided group unit may be connected to the switching hub 180, and the switching hub 180 may output an image captured by each camera 112 to the image computing server 200. Such a configuration is particularly effective when the image processing system 100 is applied to a stadium. For example, a case where a stadium has a plurality of floors and the sensor system 110 is provided for each floor can be considered. If the above-described configuration is employed in such a case, for example, an image captured by the camera 112 can be input to the image computing server 200 for each floor or for each half of the stadium. Therefore, it is possible to simplify the installation of the sensor system 110 and make the system flexible even in a place where wiring for connecting all the sensor systems 110 with one daisy chain is difficult.

  Also, the control of image processing in the image computing server 200 is switched depending on whether the number of camera adapters 120 connected to each other in a daisy chain and outputting images to the image computing server 200 is one or more. It is done. That is, control of image processing in the image computing server 200 is switched depending on whether or not the sensor system 110 is divided into a plurality of groups. When there is one camera adapter 120 that outputs an image to the image computing server 200, an image is transmitted between the sensor systems 110 connected in a daisy chain, and an image is transmitted from the terminal camera adapter 120 to the image computing server 200. The Then, the image computing server 200 generates an image of the entire circumference of the stadium. For this reason, the timing at which the image data of the entire circumference of the stadium is gathered in the image computing server 200 is synchronized. That is, if the sensor system 110 is not divided into groups, the timing at which necessary image data is prepared in the image computing server 200 can be synchronized.

  On the other hand, when there are a plurality of camera adapters 120 that output images to the image computing server 200 (the sensor system 110 is divided into groups), the daisy chain lanes (routes) of the image data at the time of transmission The delay may be different. For this reason, the image computing server 200 needs to perform subsequent image processing while checking the concentration of the image data by waiting until the image data of the entire circumference of the stadium are gathered and synchronizing each image data. .

  In the present embodiment, the sensor system 110a includes a microphone 111a, a camera 112a, a pan head 113a, an external sensor 114a, and a camera adapter 120a. Note that the configuration of the sensor system 110a is not limited to this configuration. The sensor system 110a only needs to include at least one camera adapter 120a and at least one camera 112a. For example, the sensor system 110a may be configured with one camera adapter 120a and a plurality of cameras 112a, or may be configured with one camera 112a and a plurality of camera adapters 120a. That is, at least one camera 112a and at least one camera adapter 120a in the image processing system 100a correspond to N to M (N and M are integers of 1 or more). Further, the sensor system 110a may include devices other than the microphone 111a, the camera 112a, the pan head 113a, and the camera adapter 120a. Further, the camera 112a and the camera adapter 120a may be integrally configured in the same housing. Furthermore, the front end server 230 may have at least a part of the functions of the camera adapter 120a. In the present embodiment, the configuration of the sensor systems 110b to 110z is the same as the configuration of the sensor system 110a, and thus a detailed description of the sensor systems 110b to 110z is omitted. The sensor systems 110b to 110z are not limited to the same configuration as the sensor system 110a, and may be configured differently.

  An image photographed by the camera 112a is subjected to image processing described later in the camera adapter 120a, and then transmitted to the camera adapter 120b of the sensor system 110b via the network 170a. The sound collected by the microphone 111a is also transmitted to the camera adapter 120b of the sensor system 110b via the network 170a. The sensor system 110b transmits the sound collected by the microphone 111b and the image taken by the camera 112b to the sensor system 110c together with the image and sound acquired from the sensor system 110a.

The operation described above is continued in the sensor systems 110c to 110z. As a result, the images and sounds acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z to the switching hub 180 via the network 180b, and then transmitted from the switching hub 180 to the image computing server 200.
In the present embodiment, a configuration in which the cameras 112a to 112z and the camera adapters 120a to 120z are separated is described as an example. However, as described above, these may be integrated in the same casing. In that case, the microphones 111a to 111z may be built in the cameras 112a to 112z integrated with the camera adapters 120a to 120z, or may be connected to the outside of the cameras 112a to 112z.

  Next, an example of the configuration and operation of the image computing server 200 will be described. The image computing server 200 of this embodiment processes data acquired from the sensor system 110z. The image computing server 200 includes a front end server 230, a database 250, a back end server 270, and a time server 290.

  The time server 290 has a function of distributing time and synchronization signals. The time server 290 distributes the time and the synchronization signal to the sensor systems 110a to 110z via the switching hub 180. The camera adapters 120a to 120z that have received the time and the synchronization signal Genlock the image data photographed by the cameras 112a to 112z based on the time and the synchronization signal to perform frame synchronization of the image data. That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112a to 112z. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of images shot at the same timing, and thus can suppress deterioration in the quality of the virtual viewpoint image due to a shift in shooting timing. . In the present embodiment, it is assumed that the time server 290 manages time synchronization of the plurality of cameras 112. However, the time server 290 does not necessarily have to manage time synchronization of the plurality of cameras 112. For example, each camera 112 or each camera adapter 120 may perform processing for time synchronization independently.

The front-end server 230 reconstructs a segmented transmission packet from the image data and audio data acquired from the sensor system 110z and converts the data format. The front-end server 230 writes the image data and audio data converted in data format in the database 250 in association with the camera identifier, data type, frame number, and the like.
The back-end server 270 receives a virtual viewpoint designation from the virtual camera operation UI 330. The back-end server 270 reads image data and audio data corresponding to the viewpoint received from the virtual camera operation UI 330 from the database 250 based on the viewpoint received from the virtual camera operation UI 330. The back-end server 270 performs a rendering process on the image data and audio data read from the database 250 to generate a virtual viewpoint image.

  The configuration of the image computing server 200 is not limited to this. For example, at least two of the front end server 230, the database 250, and the back end server 270 may be configured integrally. A plurality of at least one of the front-end server 230, the database 250, and the back-end server 270 may be included. In addition, a device other than the devices described above may be included in an arbitrary position in the image computing server 200. Further, the end user terminal 190 and the virtual camera operation UI 330 may have at least a part of the functions of the image computing server 200.

  The rendered virtual viewpoint image is transmitted from the back-end server 270 to the end user terminal 190. A user who operates the end user terminal 190 can view an image and view audio according to the designation of the viewpoint. That is, the back-end server 270 generates virtual viewpoint content based on images (a plurality of virtual viewpoint images) taken by a plurality of cameras 112 and virtual viewpoint information. Specifically, the back-end server 270 is based on, for example, image data of a predetermined area extracted by a plurality of camera adapters 120 from image data captured by a plurality of cameras 112, and a viewpoint specified by a user operation. To generate virtual viewpoint content. Then, the back end server 270 provides the generated virtual viewpoint content to the end user terminal 190. Details of extraction of the predetermined area by the camera adapter 120 will be described later.

  The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the viewpoint designated by the virtual camera operation UI 330. The viewpoint may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. An image corresponding to the viewpoint designated by the user from a plurality of candidates and an image corresponding to the viewpoint automatically designated by the apparatus are also included in the virtual viewpoint image. In the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described. However, the audio data may not be included in the virtual viewpoint content.

  Further, the back-end server 270 converts the virtual viewpoint image into the H.264 format. The data may be compressed and encoded by a standard technique represented by H.264 or HEVC and then transmitted to the end user terminal 190 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 190 without being compressed. The former method of performing compression encoding assumes a smartphone or a tablet as the end user terminal 190. The latter method assumes a display capable of displaying an uncompressed image as the end user terminal 190. That is, the image format can be switched according to the type of the end user terminal 190. The image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or other transmission methods may be used.

  As described above, the image processing system 100 has three functional domains: a video collection domain, a data storage domain, and a video generation domain. The video collection domain includes sensor systems 110a-110z. The data storage domain includes a database 250, a front end server 230, and a back end server 270. The video generation domain includes a virtual camera operation UI 330 and an end user terminal 190. Note that the configuration of the image processing system 100 is not limited to such a configuration. For example, the virtual camera operation UI 330 can directly acquire image data from the sensor systems 110a to 110z. However, in the present embodiment, a data storage domain is arranged between the video collection domain and the video generation domain, instead of a method of directly acquiring image data from the sensor systems 110a to 110z. Specifically, the front-end server 230 converts image data, audio data, and meta information of these data generated by the sensor systems 110a to 110z into a common schema and data type of the database 250. As a result, even if the cameras 112a to 112z of the sensor systems 110a to 110z are changed to cameras of other models, the front-end server 230 absorbs the difference between the cameras and registers the image data captured by the cameras of the other models in the database 250. I can do it. Thus, when the camera 112 is changed to a camera of another model, it is possible to reduce a possibility that the virtual camera operation UI 330 does not operate properly.

  In the present embodiment, the virtual camera operation UI 330 is accessed via the back-end server 270 without directly accessing the database 250. Common processing related to the image generation processing is performed by the back-end server 270, and processing of the difference portion of the application related to the operation UI is performed by the virtual camera operation UI 330. As a result, when developing the virtual camera operation UI 330, it is possible to focus on development of an operation device serving as a UI (user interface) and a UI function for operating a virtual viewpoint image to be generated. Further, the back-end server 270 can add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 330. This makes it possible to respond flexibly to requests from the virtual camera operation UI 330.

  As described above, in the image processing system 100, the virtual viewpoint image is generated by the back-end server 270 based on the image data captured by the plurality of cameras 112 for capturing the subject from a plurality of directions. Note that the image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be logically configured.

  In this embodiment, for example, an example of a plurality of information processing apparatuses is realized by using the camera adapters 120a to 120z. Further, the virtual camera operation UI 330 sets a viewpoint for the subject. Further, the back-end server 270 uses the image of the subject area included in the plurality of photographed images photographed from the plurality of directions, and displays the image of the subject when viewed from the viewpoint set by the virtual camera operation UI 330. Generate.

(Camera adapter)
Next, an example of functional blocks of the camera adapter 120 in the present embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a functional configuration of the camera adapter 120.
The camera adapter 120 includes a network adapter 6110, a transmission unit 6120, an image processing unit 6130, and an external device control unit 6140.
The network adapter 6110 includes a data transmission / reception unit 6111 and a time control unit 6112.

  The data transmission / reception unit 6111 performs data communication with other camera adapters 120, the front-end server 230, the time server 290, and the control station 310 via the networks 170, 291 and 310a. For example, the data transmission / reception unit 6111 outputs the foreground image and the background image separated by the foreground / background separation unit 6131 from the image taken by the camera 112 to another camera adapter 120. The camera adapter 120 that is the output destination of the foreground image and the background image is the camera adapter 120 in the next order in a predetermined order according to the processing of the data routing processing unit 6122 among the camera adapters 120 in the image processing system 100. It is. Each camera adapter 120 outputs a foreground image and a background image, thereby generating a virtual viewpoint image based on the foreground image and the background image taken from a plurality of viewpoints. There may be a camera adapter 120 that outputs a foreground image separated from a captured image and does not output a background image.

  The time control unit 6112 is based on, for example, the IEEE 1588 standard Ordinary Clock, and has a function of storing a time stamp of data transmitted to and received from the time server 290 and a function of performing time synchronization with the time server 290. Note that the protocol for time synchronization is not limited to IEEE 1588. For example, another EtherAVB standard or a unique protocol may be used. In the present embodiment, a network interface card (NIC) is used as the network adapter 6110. However, the network adapter 6110 is not limited to the NIC, and other interfaces may be used. IEEE 1588 has been updated as a standard, such as IEEE 1588-2002 and IEEE 1588-2008. The latter is also called PTPv2 (Precision Time Protocol Version 2).

The transmission unit 6120 has a function of controlling transmission of data to the switching hub 180 and the like via the network adapter 6110, and includes the following functional units.
The data compression / decompression unit 6121 has a function of compressing data transmitted / received via the data transmission / reception unit 6111 by applying a predetermined compression method, compression rate, and frame rate, and a function of decompressing the compressed data And have.

  The data routing processing unit 6122 has a function of determining the routing destination of the data received by the data transmitting / receiving unit 6111 and the data processed by the image processing unit 6130 using data held by the data routing information holding unit 6125 described later. . Further, the data routing processing unit 6122 has a function of transmitting data to the determined routing destination. The data routing processing unit 6122 preferably determines the camera adapter 120 corresponding to the camera 112 focused on the same point of sight as the camera 112 corresponding to the camera adapter 120 to which the data routing processing unit 6122 belongs as a routing destination. This is because images taken by these cameras 112 have a high frame correlation, and thus it is preferable to determine the routing destination in this way for image processing. In accordance with the determination of the routing destination by the data routing processing unit 6122 of each of the plurality of camera adapters 120, the order of the camera adapters 120 that output the foreground image and the background image in the relay format is determined.

  The time synchronization control unit 6123 conforms to the IEEE 1588 standard PTP (Precision Time Protocol) and has a function of performing processing related to time synchronization with the time server 290. Note that the protocol for performing time synchronization is not limited to PTP, and time synchronization may be performed using other similar protocols.

  The image / audio transmission processing unit 6124 has a function of creating a message for transferring image data and audio data to another camera adapter 120 or the front-end server 230 via the data transmission / reception unit 6111. The message includes image data and audio data, and meta information of each data. The meta information of this embodiment includes a time code or sequence number when an image is taken and when audio is sampled, a data type, an identifier indicating the individual of the camera 112 and the microphone 111, and the like. Note that the image data and audio data transmitted by the image / audio transmission processing unit 6124 may be compressed by the data compression / decompression unit 6121.

The image / sound transmission processing unit 6124 receives a message from another camera adapter 120 via the data transmission / reception unit 6111. Then, the image / sound transmission processing unit 6124 converts the data information fragmented into the packet size specified in the transmission protocol into image data and sound data according to the data type included in the message received from the other camera adapter 120. Restore. If the data is compressed when the data is restored, the data compression / decompression unit 6121 performs decompression processing on the data.
The data routing information holding unit 6125 has a function of holding address information for determining a transmission destination of data transmitted / received by the data transmitting / receiving unit 6111.

The image processing unit 6130 has a function of processing image data captured by the camera 112 under the control of the camera control unit 6141 and image data received from another camera adapter 120, and includes the following function units.
The foreground / background separator 6131 has a function of separating image data captured by the camera 112 into a foreground image and a background image. That is, the foreground / background separation unit 6131 extracts a predetermined area from the image data photographed by the camera 112 corresponding to the camera adapter 120 to which the foreground / background separator 6131 belongs. The predetermined area is, for example, an area of the foreground image obtained as a result of object detection for the captured image. The foreground / background separator 6131 separates the captured image into a foreground image and a background image based on the extraction result of the predetermined area. Note that an object is, for example, a subject such as a person. However, the object may be a specific person (player, manager, and / or referee, etc.), or may be an object with a predetermined image pattern, such as a ball or a goal. A moving object may be detected as the object. An image of a portion corresponding to an object of a virtual viewpoint image generated in the image processing system 100 by separating and processing a foreground image including an important object such as a person and a background area not including such an object Quality can be improved. Further, the foreground area and the background area are separated by each of the plurality of camera adapters 120, whereby the load on the image processing system 100 including the plurality of cameras 112 can be distributed. The predetermined area is not limited to the foreground image, and may be a background image, for example.

The three-dimensional information processing unit 6132 uses the foreground image separated by the foreground / background separation unit 6131, the foreground image received from the other camera adapter 120, and the imaging space information of the control station 310 to perform a three-dimensional process. It has a function to perform image information processing. For example, stereo matching is used for the three-dimensional image information processing.
The image quality adjustment unit 6133 has a function of adjusting the image quality of at least one of the foreground image and the background image separated by the foreground / background separation unit 6131. The image quality is, for example, at least one of resolution, color gradation, contrast, saturation, luminance, and brightness.

  The calibration control unit 6134 has a function of acquiring image data necessary for calibration from the camera 112 via the camera control unit 6141 and transmitting the image data to the front-end server 230 that performs arithmetic processing related to calibration. In the present embodiment, a case where the calculation process related to calibration is performed by the front-end server 230 will be described as an example. However, the node that performs the arithmetic processing is not limited to the front end server 230. For example, the calculation process may be performed in another node such as the control station 310 or the camera adapter 120 (including another camera adapter 120). In addition, the calibration control unit 6134 has a function of performing calibration during shooting (dynamic calibration) on image data acquired from the camera 112 via the camera control unit 6141 according to a preset parameter. Have.

The external device control unit 6140 has a function of controlling devices connected to the camera adapter 120, and includes the following functional units.
The camera control unit 6141 is connected to the camera 112 and has functions of controlling the camera 112, obtaining image data captured by the camera 112, providing a synchronization signal, and setting a time.

  The control of the camera 112 includes, for example, setting and reference of shooting parameters (such as setting of the number of pixels, color depth, frame rate, and white balance). Furthermore, the control of the camera 112 includes acquisition of the state of the camera 112 (during shooting, stop, synchronization, error, etc.), start and stop of shooting, and focus adjustment. When a removable lens is attached to the camera 112, the camera adapter 120 may be connected to the lens and directly adjust the lens. The camera adapter 120 may perform lens adjustment such as zooming via the camera 112.

The synchronization signal is provided by providing the camera 112 with a photographing timing (control clock) using a time synchronized with the time server 290 obtained by the time synchronization control unit 6123.
The time setting is performed by providing the time synchronized with the time server 290 obtained by the time synchronization control unit 6123 using, for example, a time code conforming to the SMPTE12M format. As a result, the time code provided from the camera control unit 6141 is added to the image data received from the camera 112. Note that the format of the time code is not limited to SMPTE12M, and other formats may be used. Further, the camera control unit 6141 may provide the time code to the image data received from the camera 112 without providing the time code to the camera 112.

The microphone control unit 6142 is connected to the microphone 111 and has a function of controlling the microphone 111, starting and stopping sound collection by the microphone 111, obtaining audio data collected by the microphone 111, and the like.
Control of the microphone 111 includes gain adjustment, acquisition of the state of the microphone 111, and the like, for example. In addition, the microphone control unit 6142 provides the microphone 111 with timing and time code for sampling the audio data. For example, time information from the time server 290 is converted into, for example, a 48 KHz word clock and supplied to the microphone 111, so that clock information serving as a timing for sampling audio data is supplied to the microphone 111.

The pan head control unit 6143 is connected to the pan head 113 and has a function of controlling the pan head 113. The control of the pan head 113 includes, for example, pan / tilt control, acquisition of the state of the pan head 113, and the like.
The sensor control unit 6144 is connected to the external sensor 114 and has a function of acquiring sensor information sensed by the external sensor 114. For example, when a gyro sensor is used as the external sensor 114, the sensor control unit 6144 acquires information indicating vibration. Prior to the processing by the foreground / background separation unit 6131, the image processing unit 6130 uses the vibration information acquired by the sensor control unit 6144 to generate image data in which vibration is suppressed. For example, assume that the camera 112 is an 8K camera. In this case, the image processing unit 6130 cuts out image data captured by the camera 112 in a size smaller than the original 8K size in consideration of vibration information, and is installed at a position adjacent to the camera 112. Alignment with 112 images is performed. As a result, the image processing unit 6130 aligns the image data captured by each camera 112 with this function provided in the camera adapter 120 even if the building vibration of the building propagates to each camera 112 at a different frequency. Can be done. As a result, the image processing unit 6130 can generate electronically image-stabilized image data, and an effect of reducing the alignment processing load for the number of cameras 112 in the image computing server 200 can be obtained. The sensor of the sensor system 110 is not limited to the external sensor 114, and the same effect can be obtained even with a sensor built in the camera adapter 120.

(Image processing unit 6130)
FIG. 3 is a diagram illustrating an example of a functional configuration of the image processing unit 6130 in the camera adapter 120.
The calibration control unit 6134 performs color correction processing, blur correction processing (electronic image stabilization processing), and the like on the image data input from the camera control unit 6141. The color correction process is a process for suppressing color variations for each camera 112. The blur correction process is a process for stabilizing the position of the image against the blur caused by the vibration of the camera 112.

An example of a functional configuration of the foreground / background separation unit 6131 will be described.
The foreground separation unit 5001 separates the foreground image from the image data captured by the camera 112 based on the result of comparison between the image data that has been aligned and the background image 5002. The background update unit 5003 generates a new background image using the background image 5002 and the image data captured by the camera 112 and subjected to alignment, and makes the background image 5002 a new background image. Update. The background cutout unit 5004 performs control to cut out a part of the background image 5002. The background cutout unit 5004 outputs the background image 5002 to the image quality adjustment unit 6133 when image quality adjustment is necessary for the cut out background image 5002, and otherwise outputs the background image 5002 to the transmission unit 6120.

Next, an example of a functional configuration of the three-dimensional information processing unit 6132 will be described.
The three-dimensional model information receiving unit 5005 receives information on the three-dimensional model generated in the front end server 230 to obtain a virtual viewpoint image from the control station 310. One of the received information is three-dimensional data representing all common coverage areas of all cameras 112 included in the image processing system 100. That is, the three-dimensional data representing all the common coverage areas of all the cameras 112 is a space obtained by performing multiplication (logical product operation) of a subject space (field of view) that can be photographed by all the cameras 112. It is. The total common coverage range depends on internal parameters (focal length, image center, lens distortion parameters, etc.) unique to the camera 112 and external parameters (rotation matrix, position vector, etc.) representing the position and orientation of the camera 112. . The total common coverage area is calculated by the control station 310. Note that if the internal parameters and external parameters of the camera 112 change during shooting, the entire common coverage area is recalculated at any time by the control station 310.

The other camera foreground receiving unit 5006 receives the foreground image separated from the image data by the other camera adapter 120.
The foreground position determination unit 5007 uses the foreground image separated by the foreground separation unit 5001 and the foreground image separated from the image data by the other camera adapter 120 using the principle of stereo matching, etc. It is determined in which area of the image space the image exists. Based on the result of this determination, the foreground position determination unit 5007 transmits the foreground image as it is to the transmission unit 6120, the process of transmitting it to the image quality adjustment unit 6133, and the process of not transmitting to either (that is, foreground image data Delete process). Details of an example of processing performed by the foreground position determination unit 5007 will be described later.

  In the present embodiment, for example, by using the stereo matching principle or the like to determine in which area of the image space the foreground image exists, by using the images of the subject included in the plurality of captured images, The position of the subject in real space is specified. In addition, by calculating the three-dimensional coordinates of the range in which the foreground image obtained by the top camera and the foreground image obtained by the own camera are matched, the subject image is included in the plurality of photographed images. The three-dimensional coordinates of the image of the region are derived.

  The image quality adjustment unit 6133 receives the foreground image and the background image determined by the foreground position determination unit 5007 and the background extraction unit 5004 to execute image quality adjustment before transmission. The image quality adjustment unit 6133 performs image quality adjustments such as resolution, color gradation, contrast, saturation, luminance, and brightness, and transmits image data that has undergone image quality adjustment to the transmission unit 6120.

(Transmission adjustment processing)
Hereinafter, an example of the transmission adjustment process of the present embodiment will be described using a specific example.
FIG. 4 is a plan view showing an example of the entire common coverage area. FIG. 4 schematically shows a state in which the field 400 and the eight cameras 112a to 112h installed at a certain height around the field 400 are viewed from above. The optical axis centers of the cameras 112a to 112h are all directed to a common gaze point 6301. Each of the cameras 112a to 112h is set to a focal length that becomes a horizontal angle of view θa to θh. Further, objects P to U exist on the field 400. Here, in order to simplify the explanation, it is assumed that the objects P to U are cubic objects. In FIG. 4, the total common coverage range of all the cameras 112 a to 112 h is the all common coverage range A. FIG. 5 is a perspective view showing an example of the entire common coverage area. FIG. 5 shows an outline of a state in which the field 400 shown in FIG. As shown in FIG. 5, the entire common coverage area A is actually a three-dimensional polyhedral area.

  All surfaces of the objects R and S existing inside the all common coverage area A are photographed by two or more cameras among the cameras 112a to 112h. For this reason, the foreground images corresponding to the objects R and S obtained from the cameras 112 a to 112 h are collected in the front end server 230. Accordingly, the front-end server 230 can generate a three-dimensional model of the objects R and S that are not missing from any 360 ° position by using the principle of stereo matching. On the other hand, the objects P, Q, T, and U that exist outside the entire common coverage area A have a surface that can only be photographed from one camera. In other words, the objects P, Q, T, and U that exist outside the entire common shooting range A do not exist in a range that can be captured by at least one of the cameras 112a to 112h. Therefore, for the objects P, Q, T, and U, only a three-dimensional model with a missing portion is generated when viewed from a certain viewpoint. In the present embodiment, each camera adapter 120 determines at any time a foreground image corresponding to an object existing outside the entire common subject area A, that is, the range in which the 3D model is definitely incomplete, and other camera adapters. 120 is not transmitted. By doing so, it is possible to reduce the data transmission load when performing daisy chain connection.

  FIG. 6 is a diagram showing only three cameras 112a to 112c on the upstream side of the daisy chain out of the eight cameras 112a to 112h shown in FIG. FIGS. 7A to 7C are diagrams illustrating examples of the foreground image obtained by separating the background image obtained from the image data photographed by the cameras 112a, 112b, and 112c, respectively. 7A to 7C, a circle is drawn on the left side surface of each object P to U in FIG. 6 in order to identify the common surface of the objects between the images. FIG. 8 is a flowchart illustrating an example of transmission adjustment processing executed by the three-dimensional information processing unit 6132. In the following, the three cameras that are continuous in the daisy chain are referred to as an upper camera, a local camera, and a lower camera from the upstream side of the daisy chain.

  First, for the most upstream camera adapter 120a in the daisy chain, the data transmitting / receiving unit 6111 transmits all the foreground images to the camera 112c, which is a subordinate camera, without executing the processing shown in the flowchart of FIG. 8 (FIG. 7). (See (a)). The foreground position determination unit 5007 in the second camera 112b is a foreground image obtained from image data photographed by the camera 112b as the foreground image and image data photographed by the camera 112b as the camera. And compare. The foreground position determination unit 5007 performs matching of corresponding points between these foreground images based on the comparison result. Note that the foreground image obtained from the image data captured by the camera 112a, which is the upper camera, is received by the other camera foreground receiving unit 5006. In the following description, the foreground image obtained from the image data captured by the top camera is referred to as the foreground image obtained by the top camera as necessary, and the foreground obtained from the image data captured by the camera itself. The image is referred to as a foreground image obtained with the own camera as necessary.

  In the present embodiment, the foreground position determination unit 5007 obtains a foreground image obtained by the top camera and a foreground image obtained from image data captured by the own camera, so that the area of the subject included in the captured image is obtained. Get the image. For example, a foreground image is an example of an image of a subject area. Further, for example, the foreground image obtained by the upper camera and the foreground image obtained by the own camera are examples of the image of the subject area included in the plurality of photographed images photographed from a plurality of directions. In addition, for example, the camera adapter 120a corresponding to the camera 112a that is the upper camera is an example of the information processing apparatus that is in the order immediately preceding the information processing apparatus in a predetermined order. In addition, the camera adapter 120c corresponding to the camera 112c, which is a subordinate camera, is an example of the information processing apparatus that becomes the next order of the information processing apparatus in a predetermined order.

  The foreground position determination unit 5007 determines a matching range between the foreground image obtained by the camera 112a that is the top camera and the foreground image obtained by the camera 112b that is the camera itself (S1001). Of these foreground images, the process of S1002 is executed for the matching range, and the process of S1006 is executed for the non-matching range. That is, when the foreground image obtained by the own camera and the foreground image obtained by the top camera include a matching range and a non-matching range, for these foreground images, the processing after S1002 and the processing after S1006 are performed. Both of these processes are performed.

  In step S1002, the foreground position determination unit 5007 calculates the three-dimensional coordinates of the matching range between the foreground image obtained by the top camera and the foreground image obtained by the own camera using the principle of stereo matching (S1002).

Next, based on the three-dimensional coordinates obtained in S1002, the foreground position determination unit 5007 determines that the matching range between the foreground image obtained by the superimposing camera and the foreground image obtained by the own camera is the entire common coverage area. It is determined whether it is inside A (S1003). The entire common coverage area A is received from the control station 310.
As a result of this determination, if the matching range between the foreground image obtained by the superimposing camera and the foreground image obtained by the own camera is inside the all common object range A (Yes in S1003), the foreground position determination unit 5007 performs the following processing. That is, the foreground position determination unit 5007 determines to transmit a matching range between the foreground image obtained by the upper camera and the foreground image obtained by the own camera to the camera adapter 120c corresponding to the camera 112c that is the lower camera. (S1004).

  In this embodiment, the foreground position determination unit 5007 transmits an image of the subject area included in the photographed image when there is a subject area in the common area of the subject range in at least two photographing apparatuses in S1004. .

  On the other hand, if the matching range between the foreground image obtained with the top camera and the foreground image obtained with the own camera is not within the all common subject range A (No in S1003), the foreground position determination unit 5007 Perform the following process. That is, the foreground position determination unit 5007 transmits an image in the foreground image obtained by the top camera if the range matching the foreground image obtained by the own camera is not inside the all common coverage area A. It decides not to do (S1005). The foreground position determination unit 5007 transmits an image in the range when the range of the foreground image obtained by the own camera and the foreground image obtained by the top camera is not inside the all-common captured area A. Then, it determines (S1005). However, the foreground position determination unit 5007 limits the transmission of the image in the range to the camera adapter 120c corresponding to the camera 112c that is the underlying camera (S1005).

  At this time, the range of the foreground image obtained by the own camera that matches the foreground image obtained by the upper camera and that is not within the entire common coverage area A may be not transmitted. However, in the present embodiment, the image is transmitted up to the camera adapter 120c corresponding to the camera 112c that is the underlying camera. This is because the matching search accuracy of the corresponding points of the foreground image in the underlying camera can be maintained. However, the image is not transmitted to the camera adapters 120d to 120h corresponding to the cameras 112d to 112h on the downstream side of the camera 112c that is the underlying camera. Specifically, the foreground position determination unit 5007 of the camera adapter 120c determines not to transmit the image to the camera adapter 120d. Note that in step S1005, the camera adapter 120 may compress and resize and transmit a foreground image that is not inside the entire common coverage area A. In this way, by limiting the transmission by reducing the data amount of the foreground image that is not inside the entire common coverage area A, the subject exists in a wider range while suppressing the transmission load of the system. A virtual viewpoint image to be generated can be generated.

  In this embodiment, the foreground position determination unit 5007 controls transmission of data including the image of the subject area included in the captured image based on the position of the subject in the real space in S1003 to S1005. In S1005, the foreground position determination unit 5007 does not transmit the image of the subject area included in the captured image, or restricts the transmission destination of the image of the subject area included in the captured image. Is selected according to the photographing device that photographed the photographed image.

  On the other hand, for the non-matching range between the foreground image obtained by the camera 112a which is the upper camera and the foreground image obtained by the camera 112b which is the own camera, the processing of S1006 is performed. In other words, the foreground position determination unit 5007 adds 1 to the non-matching count n for the foreground image obtained by the camera 112a as the upper camera and the foreground image obtained by the camera 112b as the own camera (S1006). The initial value of the non-matching count n is 0 (zero).

  In the present embodiment, for example, an example of the count value is realized by the number of non-matching. Further, in S1006, the foreground position determination unit 5007 determines that the image of the subject area included in the captured image does not match the image of the subject area included in the captured image different from the captured image. Update the count value for.

  Next, the foreground position determination unit 5007 determines the number n of non-matching for a non-matching range between the foreground image obtained by the camera 112a that is the top camera and the foreground image obtained by the camera 112b that is the camera itself (S1007). ). Of the non-matching range between the foreground image obtained by the camera 112a which is the top camera and the foreground image obtained by the camera 112b which is the own camera, the processing of S1008 is performed for the range where the non-matching count n is 2. Is called. On the other hand, of the non-matching range between the foreground image obtained by the camera 112a which is the upper camera and the foreground image obtained by the camera 112b which is the own camera (the range where the non-matching number n is not 2). In S1004, the process is performed.

In step S1008, the foreground position determination unit 5007 performs the following process. That is, the foreground position determination unit 5007 has a non-matching count n of 2 in a non-matching range between the foreground image obtained by the camera 112a that is the top camera and the foreground image obtained by the camera 112b that is the camera itself. It is determined not to transmit the range (S1008).
In the present embodiment, for example, S1008 is an example of determining that the image is not transmitted when the count value for the unmatched image is larger than a predetermined value. Note that the camera adapter 120 may be configured to limit the transmission of all foreground images that do not match between the foreground image obtained by the top camera and the foreground image obtained by the own camera without performing the processing of S1006 and S1007.

  On the other hand, when the process proceeds to S1004, the foreground position determination unit 5007 performs the following process. That is, the foreground position determination unit 5007 is a range in which the number n of non-matching is not 2 out of the non-matching range between the foreground image obtained by the camera 112a as the upper camera and the foreground image obtained by the camera 112b as the own camera. Is determined to be transmitted (S1004). Here, since only two cameras 112a and 112b are processed, the non-matching count n does not become two. Therefore, the entire non-matching range between the foreground image obtained by the camera 112a as the upper camera and the foreground image obtained by the camera 112b as the own camera is transmitted.

  In the present embodiment, for example, S1004 is an example of determining that the image is to be transmitted when the count value for the unmatched image is smaller than a predetermined value. In addition, for example, S1004, S1007, and S1008 realize an example of controlling transmission of data including the image based on the count value for the unmatched image. Also, for example, through S1001 to S1008, based on whether or not the image of the subject area included in the plurality of captured images is matched, data including the image of the subject area included in the captured image is transmitted. An example of controlling is realized.

FIG. 9 is a diagram for explaining an example of an object transmitted to the underlying camera.
The upper part of FIG. 9 shows the result of executing the processing of S1001 to S1007 on the foreground image shown in FIGS. 7 (a) and 7 (b). 9 corresponds to FIG. 7A, and the right side of the upper stage in FIG. 9 corresponds to FIG. 7B. In the upper part of FIG. 9, the range of the foreground image that is determined to be inside the all-common coverage area A and determined to be transmitted to the camera 112c that is the underlying camera is a blackened range. In addition, in the upper part of FIG. 9, the range of the foreground image that is determined to be outside the entire common coverage range A and is transmitted only to the camera 112c that is the underlying camera is a vertical stripe range. In the upper part of FIG. 9, the foreground image that is not transmitted is in a range surrounded by a broken line. In the upper part of FIG. 9, the range of the foreground image transmitted when the non-matching count n is 1 is a hatched range.

  In the subsequent third camera 112c, the result of executing the processing according to the flowchart of FIG. 8 in the same manner as in the case where the upper camera is the camera 112b, the own camera is the camera 112c, and the own camera is the camera 112b. Shown below. The lower left side of FIG. 9 corresponds to FIG. 7B, and the lower right side of FIG. 9 corresponds to FIG. 7C. The surface coloring in the lower part of FIG. 9 is the same as the surface coloring in the upper part of FIG. Here, on the left side of the lower part of FIG. 9, the surface of the object Q which is a broken diagonal line is only reflected in the camera 112b, so the non-matching count n is 2, and the surface determined to be non-transmitted. is there.

  At the time when the second camera 112b is its own camera, the object U is an object that is outside the entire common subject range A but is transmitted only to the camera 112c, which is a subordinate camera. The range of a part of the foreground image of the object U (the range of the vertical stripe on the right side of the upper stage in FIG. 9) is the foreground image obtained by the camera 112c which is the own camera when the third camera 112c is the own camera. Is not matched and determined as non-transmission. Accordingly, a partial range of the foreground image of the object U (the vertical stripe region on the right side in the upper part of FIG. 9) is not transmitted after the fourth camera. If there is another object or obstacle in front of the foreground image obtained by the camera 112c which is the own camera, the object U is not reflected, and the foreground image obtained by the foreground camera and the foreground image are obtained. Suppose that the corresponding points could not be matched. Also in such a case, the foreground image of the object U is not transmitted to the subsequent cameras 112d to 112h.

  Here, attention is paid to the left side of the lower part of FIG. Only the foreground images of the objects R and S existing inside the entire common coverage area A remain as transmission targets. On the other hand, the foreground images of the object Q, T, U that are located outside the entire common object range A are not transmitted. In this way, the foreground images are transmitted from the upstream to the downstream between the three cameras 112 that are consecutive in the daisy chain, and the foreground images obtained by the front and rear cameras are compared, and the entire common coverage range A is obtained. It is determined whether or not there is a non-matching count n. In this way, it is possible to select and transmit only the foreground image of the object existing inside the all common coverage area A. As for the foreground image obtained by the first camera 112a in the daisy chain, the camera 112a is used as its own camera, and the camera 112h located at the end of the daisy chain is used as the upper camera. The position can be determined.

  As described above, in the present embodiment, among the foreground image ranges obtained by two continuous cameras (the own camera and the top camera) in the daisy chain connection, they match each other and The range inside is transmitted to the downstream camera (underlying camera). Further, of the foreground images obtained by two consecutive cameras (the own camera and the top camera) that match each other, the range outside the all common coverage area A is transmitted as follows. Determine whether or not. In other words, the range of the foreground image obtained by the own camera is transmitted, and the range of the foreground image obtained by the upper camera is not transmitted. Further, the range of the foreground image where the non-matching count n is 1 is transmitted to the downstream camera (the subordinate camera), but the range of the foreground image where the non-matching count n is 2 is transmitted to the downstream camera. Do not transmit to (underlying camera). Therefore, when generating a subject image (three-dimensional model) using images taken from a plurality of viewpoints, the foreground image is accurately extracted, and the transmission load is optimized while optimizing the image quality according to the position of the foreground image. It becomes possible to reduce.

<Second Embodiment>
Next, a second embodiment will be described. In the first embodiment, the range of the foreground image obtained by two consecutive cameras (the own camera and the top-facing camera) is a range that is outside the all common subject range A among the ranges that match each other, Does not transmit the range of the foreground image obtained with the top camera. In contrast, in the present embodiment, transmission is performed after reducing the amount of data in the range of such foreground images. As described above, the present embodiment and the first embodiment are based on the entire common subject range A in the foreground images obtained by two consecutive cameras (the own camera and the top camera). The processing for the outer range is mainly different. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS.

  FIG. 10 is a flowchart for explaining an example of transmission adjustment processing executed by the three-dimensional information processing unit 6132. FIG. 11 is a diagram illustrating an example of an object transmitted to the underlying camera. 10, steps that perform the same processing as in the flowchart of FIG. 8 are denoted by the same reference numerals as those in FIG. Further, the surface coloring in FIG. 11 is the same as the surface coloring in FIG. Therefore, detailed description thereof will be omitted.

  In the present embodiment, the foreground position determination unit 5007 performs the following processing. In other words, the foreground position determination unit 5007 reduces the image quality of images in the range outside the all-common captured area A out of the range in which the foreground image obtained by the own camera matches the foreground image obtained by the top camera. Then, it is determined that the data is to be transmitted (S2005). Here, reducing the image quality is executed by the image quality adjustment unit 6133. In addition, lowering the image quality is realized by, for example, lowering the resolution, lowering the number of color gradations, lowering the contrast, and making the color image monochrome. In S2005 as well as S1005 in FIG. 8, the foreground position determination unit 5007 has a range in which the foreground image obtained by the own camera matches the foreground image obtained by the top camera within the all-common captured area A. If not, it is determined to transmit the image in the range. This is to maintain the matching search accuracy of the corresponding points of the foreground image in the lower camera. However, the image in the range is not transmitted to the camera adapter 120 corresponding to the camera 112d on the downstream side of the underlying camera.

  In the present embodiment, the foreground position determination unit 5007, in S2005, if there is no subject area in the common area of the subject range in at least two photographing devices, the foreground position determination unit 5007 It is determined to transmit with a reduced amount of image data. In S2005, when there is no subject area in the common area of the subject range in at least two photographing apparatuses, it is determined to limit the transmission destination of the image of the subject area included in the photographed image.

  FIG. 11 shows a foreground image obtained by transmitting the foreground image shown in FIG. 7 to the front-end server 230 while applying the processing according to the flowchart of FIG. Here, the front end server 230 shows the result of applying the processing according to the flowchart of FIG. 10 to the image obtained by the first camera 112a using the foreground image obtained by the camera 112h located at the end ( (Refer to the upper left side of FIG. 11). In FIG. 11, the range indicated by the dot pattern is a range that is transmitted after the image quality is lowered. As shown in FIG. 11, the foreground images of the objects T and U that are outside the all common coverage area A but whose corresponding points can be matched and whose three-dimensional coordinates can be calculated are transmitted with reduced image quality. Is done.

  On the other hand, the upper and right surfaces of the object T in FIG. 4 are photographed only from the camera 112g. Also, the surface of the object U on the right side in FIG. 4 is photographed only from the camera 112f. For this reason, the front-end server 230 cannot generate a three-dimensional model of these surfaces. That is, the front-end server 230 has a low-quality model that is only about half of the objects T and U. Such image data cannot be said to be complete virtual viewpoint data corresponding to 360 ° free camera work. However, in the generation of virtual viewpoint content, it can be used to represent, for example, front blur and rear blur (largely blurred images existing before and after the main object that is the attention point of the content on the image) and the like. . If the image quality of the main object is maintained and the surrounding objects are blurred, the viewer may feel uncomfortable (even if a part of the model is missing) or the image quality will be degraded. Can be suppressed.

  On the other hand, in the upper part of FIG. 11, the surfaces shown in FIG. Since these planes are reflected in only one of the cameras 112a and 112b, the non-matching count n is 2, and it is determined that the transmission is not performed. Even if these planes are transmitted to the front-end server 230, they cannot be three-dimensional modeled by stereo matching. For this reason, in order to improve the reduction effect of the transmission load, it is preferable not to transmit as in the first embodiment. Thus, the objects R, S, T, and U as shown in the lower part of FIG. However, as described above, the objects T and U are transmitted with reduced image quality.

  As described above, in the present embodiment, the foreground images of the objects T and U that can be matched and can calculate the three-dimensional coordinates even when they are outside the entire common coverage area A are transmitted with reduced image quality. To do. Therefore, in addition to the effects described in the first embodiment, there is an effect that it is possible to widen the expression range of the virtual viewpoint content while reducing the transmission load.

  In the present embodiment, the case where the transmission load is reduced by adjusting the image quality has been described as an example (see S2005). However, if the data amount of the image to be transmitted can be reduced, it is not always necessary to lower the image quality. For example, the transmission load may be reduced by increasing the compression rate in the data compression / decompression unit 6121 in the range of the corresponding foreground image. Further, the transmission frame rate of only the range of the corresponding foreground image may be lowered. Further, the transmission load may be reduced by combining at least two of these controls.

  Further, in the first and second embodiments, the case where the range of the foreground image that is inside the all common subject range A is determined has been described as an example. However, it is not always necessary to determine the position of the foreground image in this way. For example, in order to absorb errors when calculating the three-dimensional coordinates, the range of the foreground image inside the three-dimensional space obtained by enlarging the all common object range A at an arbitrary ratio may be determined. On the other hand, in order to maintain high image quality in a more limited range or to increase the effect of reducing the transmission load, it is possible to determine a foreground image inside a part of the three-dimensional space of the entire common coverage range A. good. Further, depending on the setting of the completion rate required for the three-dimensional model of the object, that is, what percentage of the three-dimensional model of the surface of the object is to be generated, a common number of cameras 112 that are not the total number are common The coverage area may be calculated. In this case, the range of the foreground image that is inside the common field of view of the certain number of cameras 112 or more is determined. Furthermore, the image quality adjustment and the compression rate of the foreground image may be adjusted stepwise in accordance with the common image coverage (how many cameras are in common image coverage) and the distance from the most desired range.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  110a to 110z: sensor system, 112a to 112z: camera, 120a to 120z: camera adapter, 170a to 170y: network

Claims (27)

Acquisition means for acquiring an image of a subject area included in the captured image;
An information processing system comprising: control means for controlling transmission of data including an image of a region of the subject included in the photographed image based on the position of the subject. Further comprising specifying means for specifying the position of the subject in real space;
The acquisition means acquires images of a subject area included in a plurality of captured images captured from a plurality of directions,
The information processing system according to claim 1, wherein the specifying unit specifies a position of the subject in real space using images of the subject included in the plurality of captured images.   The specifying means derives the three-dimensional coordinates of the image of the subject area using the images of the subject included in the plurality of captured images, and based on the derived three-dimensional coordinates of the image of the subject area, The information processing system according to claim 2, wherein the position of the subject in real space is specified.   The control unit is configured to capture the image based on whether or not the subject is located in a common area of the field of view in at least two of the plurality of photographing units for photographing the scene from a plurality of directions. The information processing system according to claim 1, wherein transmission of data including an image of the subject area included in the image is controlled.   The control means is included in the photographed image when there is a subject area in the common area of the subject range of at least two photographing means among the plurality of photographing means for photographing the object scene from a plurality of directions. 5. The information processing system according to claim 4, wherein it is determined to transmit an image of the subject area.   The control means is included in the photographed image when there is no subject area in the common area of the subject area of at least two photographing means among the plurality of photographing means for photographing the object scene from a plurality of directions. 6. The information processing system according to claim 4, wherein it is determined not to transmit an image of the subject area.   The control means is included in the photographed image when there is no subject area in the common area of the subject area of at least two photographing means among the plurality of photographing means for photographing the object scene from a plurality of directions. 6. The information processing system according to claim 4, wherein the image of the subject area is determined to be transmitted with a reduced data amount of the image.   The information processing system according to claim 7, wherein reducing the data amount of the image is performed by changing at least one of an image quality, a frame rate, and a compression rate of the image.   The control means is included in the photographed image when there is no subject area in the common area of the subject area of at least two photographing means among the plurality of photographing means for photographing the object scene from a plurality of directions. 6. The information processing system according to claim 4, wherein it is determined to limit a transmission destination of an image of the subject area.   The control means is included in the photographed image when there is no subject area in the common area of the subject area of at least two photographing means among the plurality of photographing means for photographing the object scene from a plurality of directions. Either not transmitting the image of the subject area or restricting the transmission destination of the image of the subject area included in the captured image is selected according to the imaging unit that captured the captured image. The information processing system according to claim 4 or 5, wherein The acquisition means acquires images of a subject area included in a plurality of captured images captured from a plurality of directions,
The control means controls transmission of data including the image of the subject area included in the captured image based on whether or not the image of the subject area included in the plurality of captured images is matched. The information processing system according to any one of claims 1 to 10, wherein the information processing system is characterized in that: When the image of the subject area included in the photographed image does not match the image of the subject area included in the photographed image different from the photographed image, the update unit further updates the count value for the unmatched image. ,
The information processing system according to claim 11, wherein the control unit controls transmission of data including the image based on the count value for the non-matching image.   The information processing system according to claim 12, wherein the control unit determines that the image is not transmitted when the count value for the unmatched image is larger than a predetermined value.   The information processing system according to claim 12 or 13, wherein the control unit determines to transmit the image when the count value for the unmatched image is smaller than a predetermined value.   The information processing system according to claim 12, further comprising a plurality of information processing apparatuses each having the acquisition unit, the control unit, and the update unit. The control means included in the information processing apparatus transmits data including an image of a subject area included in the photographed image to the information processing apparatus that is in the next order of the information processing apparatus in a predetermined order. Control
The acquisition unit included in the information processing apparatus is configured so that an image of the area of the subject included in a photographed image photographed by a photographing unit connected to the information processing apparatus and the information processing apparatus 1 in a predetermined order. An image of the area of the subject transmitted based on control by the control means of the information processing apparatus in the previous order, and
The update unit included in the information processing apparatus includes an image of the area of the subject included in a photographed image photographed by a photographing unit connected to the information processing apparatus, and one of the information processing apparatuses in a predetermined order. The count value for the image is updated when the image of the subject area transmitted based on the control by the control unit of the information processing apparatus in the previous order does not match. 15. The information processing system according to 15.   The information processing system according to claim 1, further comprising a plurality of information processing apparatuses each having the acquisition unit and the control unit. The control means included in the information processing apparatus transmits data including an image of a subject area included in the photographed image to the information processing apparatus that is in the next order of the information processing apparatus in a predetermined order. Control
The acquisition unit included in the information processing apparatus is configured so that an image of the area of the subject included in a photographed image photographed by a photographing unit connected to the information processing apparatus and the information processing apparatus 1 in a predetermined order. 18. The information processing system according to claim 17, wherein an image of the area of the subject transmitted based on control by the control unit of the information processing apparatus in the previous order is acquired.   The control means included in the information processing apparatus has a subject area in a common area of a subject range in at least two photographing means among a plurality of photographing means for photographing a scene from a plurality of directions. The image of the subject area included in the photographed image photographed by the photographing means connected to the information processing apparatus is transferred to the information processing apparatus that is in the order after the information processing apparatus in a predetermined order. 19. The information processing system according to claim 16, wherein the information processing system is determined to be transmitted.   The information processing system according to claim 15, wherein the plurality of information processing apparatuses are daisy chain connected. Setting means for setting a viewpoint for the subject;
Generation means for generating an image of the subject when viewed from the viewpoint set by the setting means, using images of the area of the subject included in the plurality of captured images taken from a plurality of directions; The information processing system according to any one of claims 1 to 20, wherein An acquisition step of acquiring an image of a subject area included in the captured image;
And a control step of controlling transmission of data including an image of the region of the subject included in the captured image based on the position of the subject.   A program that causes the information processing system according to any one of claims 1 to 21 to function as a computer. In a system for generating a virtual viewpoint image using photographed images photographed by a plurality of photographing devices, the device transmits a photographed image photographed by a first photographing device among the plurality of photographing devices,
Means for obtaining a photographed image photographed by the first photographing device;
Control means for restricting transmission of an image including the predetermined subject when the predetermined subject included in the captured image acquired by the acquisition means does not exist in the imaging range of the second imaging device among the plurality of imaging devices. And a device characterized by comprising:   The apparatus according to claim 24, wherein the control unit does not transmit an image including the predetermined subject.   The control means transmits an image including the predetermined subject to another device that transmits a photographed image photographed by a third photographing device among the plurality of photographing devices, and generates the virtual viewpoint image. 25. The device of claim 24, wherein the device is not transmitted to the device.   25. The apparatus according to claim 24, wherein the control means transmits the image including the predetermined subject with a reduced image quality.