JP2019140483A - Image processing system, image processing system control method, transmission device, transmission method, and program - Google Patents

Abstract

To enhance foreground reproducibility in a virtual viewpoint image even in a situation where the band of a transmission line is limited.SOLUTION: An image processing system which transmits an image taken by plural cameras via a predetermined transmission line and generates a virtual viewpoint image comprises: means which derives the number of cameras taking a predetermined object; and means which, in response to the number of the cameras taking the predetermined object, sets the quality of a front ground including the predetermined object.SELECTED DRAWING: Figure 12

Description

  The present invention relates to an image processing system including a plurality of cameras for photographing a subject from a plurality of directions.

  In recent years, attention has been paid to a technique for installing a plurality of cameras at different positions, performing synchronous shooting from multiple viewpoints, and generating virtual viewpoint content using a plurality of viewpoint images obtained by the shooting. According to the technology for generating virtual viewpoint content from the plurality of viewpoint images, for example, it is possible to view a highlight scene of soccer or basketball from various angles, so that a high sense of reality can be given to the user.

  On the other hand, when an image (video) acquired by a camera is transmitted over a network, it is necessary to control the code amount of the video so as not to exceed the network bandwidth. Here, in Patent Document 1, when a plurality of surveillance cameras are connected to the same network and video is transmitted, a frame is set so that the total code amount of the plurality of surveillance camera videos does not exceed a predetermined band (capacity). A method for controlling the rate is disclosed. In Patent Document 2, when a plurality of surveillance cameras are connected to the same network and video is transmitted, a video signal including an important area such as a target subject (that is, a camera that captures the important area) is given higher priority. A method of controlling the frame rate by setting is disclosed.

JP 2004-32680 A JP 2007-221367 A

  On the other hand, one of the important factors in the image quality of virtual viewpoint content is the reproducibility of the foreground (for example, a person). Regarding foreground reproducibility, it is important how many cameras at different positions can capture the subject. This is related to mainly using video from a camera located near the virtual viewpoint when generating the virtual viewpoint content. The closer the camera position is to the virtual viewpoint, the more reproducible foreground is. It is because it is obtained.

  In view of the foreground reproducibility, the method described in Patent Document 1 uniformly controls the frame rate of cameras on the same network, so that virtual viewpoint content can be generated for frames determined not to be transmitted. It cannot be used. In addition, as described above, the method described in Patent Document 2 sets a higher priority for a camera that captures an important area. However, in the case of shooting around a single gazing point, the camera that captures the important area. Since there are many, it becomes difficult to set priority. For this reason, there is a possibility that a foreground with high reproducibility cannot be acquired by either of the method described in Patent Document 1 and the method described in Patent Document 2.

  The present invention has been made in view of the above-described conventional problems, and it is an object of the present invention to improve the foreground reproducibility in a virtual viewpoint image even in a situation where the bandwidth of the transmission path is limited.

  In order to achieve the above object, the present invention provides an image processing system that transmits images taken by a plurality of cameras via a predetermined transmission path and generates a virtual viewpoint image. Camera number deriving means for deriving the number, and image quality setting means for setting the image quality of a foreground image including the predetermined object in accordance with the number of cameras shooting the predetermined object. To do.

  According to the present invention, foreground reproducibility can be improved in a virtual viewpoint image even in a situation where the bandwidth of the transmission path is limited.

It is a block diagram which shows the structure of an image processing system. It is a functional block diagram of a camera adapter. It is a functional block diagram of an image processing part. It is a functional block diagram of a front end server. It is a functional block diagram of a data input control part. It is a functional block diagram of a database. It is a functional block diagram of a back end server. It is a conceptual diagram for demonstrating a foreground image and a background image. It is a flowchart which shows the procedure of code amount control processing. It is a figure for demonstrating a visual field duplication map. It is a figure for demonstrating the relationship between a quantization parameter, a compression rate, and a visual field overlap degree. It is a flowchart which shows the procedure of the process which determines the quantization parameter of an initial frame. It is a flowchart which shows the procedure of code amount control processing. It is a figure for demonstrating the relationship between a quantization parameter and a visual field overlap degree.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration of an image processing system according to the present embodiment. The image processing system 100 is a system that performs shooting and sound collection by installing a plurality of cameras and microphones in facilities such as a stadium (stadium) and a concert hall. The image processing system 100 includes sensor systems 110a to 110z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

  The controller 300 includes a control station 310 and a virtual camera operation UI (User Interface) 330. The control station 310 performs operation state management, parameter setting control, and the like through the networks 310a to 310c, 180a, and 180b, and the daisy chain 170a to 170y for each block constituting the image processing system 100. Note that these networks may be Ethernet (registered trademark) GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, or may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. Moreover, it is not limited to these, Other types of networks may be sufficient.

  First, a process of transmitting 26 sets of images and sounds of the sensor systems 110a to 110z from the sensor system 110z to the image computing server 200 will be described. In addition, in this embodiment, when there is no special description, 26 sets of systems from the sensor system 110a to the sensor system 110z are not distinguished, and are described as the sensor system 110. Similarly, the devices in each sensor system 110 are also referred to as a microphone 111, a camera 112, a pan head 113, an external sensor 114, and a camera adapter 120 unless otherwise specified. In the present embodiment, the number of sensor systems is described as 26 sets. However, this is just an example, and the number of sensor systems is not necessarily limited to this.

  Furthermore, in this embodiment, the term “image” includes the concept of a still image and a moving image unless otherwise specified. That is, in the image processing system 100 of the present embodiment, both still images and moving images can be processed. In addition, in this embodiment, an example in which a virtual viewpoint image and a virtual viewpoint sound are included as the virtual viewpoint content provided by the image processing system 100 will be mainly described. However, the virtual viewpoint content provided by the image processing system 100 is described. Is not necessarily limited to this. Therefore, for example, audio may not be included in the virtual viewpoint content. For example, the sound included in the virtual viewpoint content may be a sound collected by a microphone closest to the virtual viewpoint. In addition, in this embodiment, for simplification of explanation, description about sound is partially omitted, but both image and sound are basically processed.

  As shown in FIG. 1, in the image processing system 100 of the present embodiment, each sensor system 110 has one camera 112. That is, the image processing system 100 includes a plurality of cameras for photographing the subject from a plurality of directions.

  The plurality of sensor systems 110 (i.e., sensor systems 110a-110z) are connected by a daisy chain. In this way, by connecting through a daisy chain, the number of connected cables can be reduced and wiring work can be achieved in the increase in the capacity of image data due to the increase in resolution and frame rate of captured images to 4K, 8K, etc. It can save labor.

  In addition, as a form which connects the sensor system 110, it is not necessarily limited to this. Therefore, for example, each sensor system 110a-110z may be connected to the switching hub 180, and a star-type network configuration that performs data communication between the sensor systems 110 via the switching hub 180 may be employed.

  Further, in FIG. 1, the configuration in which all of the sensor systems 110a to 110z are cascade-connected so as to be connected in a daisy chain is shown, but the configuration is not necessarily limited thereto. Thus, for example, the sensor system 110 may be divided into several groups and daisy chain connected in units of the divided groups. In that case, the camera adapter 120 that is the end of the divided group may be connected to the switching hub 180 and input an image to the image computing server 200.

  Such a configuration is particularly effective in a stadium. For example, it is assumed that the stadium is composed of a plurality of floors and the sensor system 110 is provided for each floor. In this case, the sensor system 110 having the above-described configuration allows input to the image computing server 200 for each floor or for each half of the stadium. Thus, even in a place where wiring for connecting all the sensor systems 110 with one daisy chain is difficult, the installation can be simplified and the system can be made flexible.

  Further, in a configuration connected in a daisy chain, image processing in the image computing server 200 depends on whether the number of camera adapters 120 that output an image to the image computing server 200 is one or two or more. Switch the control. That is, in the image processing system 100, the image processing control in the image computing server 200 is switched depending on whether the sensor system 110 is divided into a plurality of groups.

  Here, when there is one camera adapter 120 that outputs an image, the entire image of the stadium is generated while transmitting the image through a daisy chain connection. Timing is synchronized. That is, if the sensor system 110 is not divided into groups, synchronization can be achieved. However, when there are a plurality of camera adapters 120 that output images (that is, when the sensor system 110 is divided into a plurality of groups), it is assumed that the delay differs depending on the lane (path) of each daisy chain. Therefore, when the sensor system 110 is divided into a plurality of groups, the image computing server 200 waits until the image data of the entire circumference is gathered, and the synchronization control is performed to confirm the collection of the image data. The subsequent image processing is executed.

  The sensor system 110a includes a microphone 111a, a camera 112a, a pan head 113a, an external sensor 114a, and a camera adapter 120a. Note that the configuration of the sensor system 110a is not necessarily limited to this configuration, and it is sufficient that the sensor system 110a has at least one camera adapter 120a and one camera 112a or one microphone 111a. Further, for example, the sensor system 110a may be configured to have one camera adapter 120a and a plurality of cameras 112a, or may be configured to have a plurality of camera adapters 120a and one camera 112a. Good. That is, the plurality of cameras 112 and the plurality of camera adapters 120 in the image processing system 100 are configured by N to M (N and M are both integers of 1 or more).

  Furthermore, the sensor system 110a may include devices other than the microphone 111a, the camera 112a, the pan head 113a, and the camera adapter 120a. The camera 112a and the camera adapter 120a may be integrated. In addition, the front end server 230 may have at least a part of the functions of the camera adapter 120.

  The sensor system 110a has been described above. In the present embodiment, the sensor systems 110b to 110z have the same configuration as that of the sensor system 110a, and thus the description thereof is omitted. The configuration of the sensor systems 110b to 110z is not limited to the same configuration as the sensor system 110a, and each sensor system 110 may have a configuration different from the sensor system 110a.

  In the above-described configuration, the sensor system 110a performs image processing described later on the camera adapter 120a on the sound collected by the microphone 111a and the image captured by the camera 112a. When the image processing is performed, the sensor system 110a transmits it to the camera adapter 120b of the sensor system 110b via the daisy chain 170a. Similarly, the sensor system 110b performs image processing on the collected sound and the captured image, and transmits them together with the image and sound acquired from the sensor system 110a to the sensor system 110c.

  By repeating these operations in order, the images and sounds acquired by the sensor systems 110a to 110z are transmitted to the switching hub 180 via the sensor systems 110z to 180b and further to the image computing server 200. .

  In the present embodiment, the cameras 112a-112z and the camera adapters 120a-120z are separated from each other, but may be integrated in the same casing. In that case, the microphones 111a to 111z may be incorporated in the integrated camera 112 or connected to the outside of the camera 112.

  Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 of this embodiment processes data acquired from the sensor system 110z. The image computing server 200 includes a front-end server 230, a database 250 (hereinafter also referred to as DB), a back-end server 270, and a time server 290.

  The time server 290 has a function of distributing the time and the synchronization signal, and distributes the time and the synchronization signal to the sensor systems 110a to 110z via the switching hub 180. The camera adapters 120a to 120z that have received the time and the synchronization signal make the cameras 112a to 112z Genlock based on the time and the synchronization signal, and synchronize images (frames). That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of images (hereinafter also referred to as “captured images”) captured at the same timing. Can be suppressed. In this embodiment, the time server 290 manages the time synchronization of the plurality of cameras 112. However, the present invention is not necessarily limited to this, and each camera 112 or each camera adapter 120 performs processing related to time synchronization independently. May be executed.

  The front-end server 230 reconstructs a segmented transmission packet from the image and sound acquired from the sensor system 110z and converts the data format. Then, data is written in the database 250 according to the camera identifier, data type, and frame number.

  The database 250 creates a database table so that the data received from the front end server 230 can be accessed. Further, it is determined whether the data requested from the back-end server 270 is stored in the cache, the primary storage, or the secondary storage, and the data is read from the storage destination and transmitted to the back-end server 270.

  The back-end server 270 receives the designation of the viewpoint from the virtual camera operation UI 330, reads the corresponding image and audio data from the database 250 based on the received viewpoint, performs rendering processing, and generates a virtual viewpoint image.

  Note that the configuration of the image computing server 200 is not necessarily limited to this. Therefore, for example, at least two of the front end server 230, the database 250, and the back end server 270 may be configured integrally. Further, a plurality of at least one of the front-end server 230, the database 250, and the back-end server 270 may be included. Furthermore, an apparatus other than the above-described apparatuses may be included in any position in the image computing server 200. In addition, at least part of the functions of the image computing server 200 may be configured to be included in the end user terminal 190 and the virtual camera operation UI 330.

  The rendered image is transmitted from the back-end server 270 to the end user terminal 190, and the user can operate the end user terminal 190 to perform image browsing and audio viewing according to the viewpoint designation. That is, the back-end server 270 generates virtual viewpoint content based on captured images (multi-viewpoint images) captured by a plurality of cameras 112 and viewpoint information. More specifically, the back-end server 270 performs, for example, virtual image processing based on image data of a predetermined area extracted from images captured by the plurality of cameras 112 by the plurality of camera adapters 120 and a viewpoint designated by a user operation. Generate viewpoint content. Then, the back end server 270 provides the generated virtual viewpoint content to the end user terminal 190. The details of the process of extracting the image data of the predetermined area by the camera adapter 120 will be described later and will be supplemented.

  In the present embodiment, the virtual viewpoint content includes a virtual viewpoint image (that is, an image representing appearance at a specified viewpoint) as an image acquired when a subject is photographed from a virtual viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of the image analysis. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. In addition, an image corresponding to the viewpoint designated by the user from a plurality of candidates and an image corresponding to the viewpoint automatically designated by the apparatus are also included in the virtual viewpoint image.

  As a supplement, in the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described. However, audio data may not necessarily be included. Further, the back-end server 270 converts the virtual viewpoint image to H.264. The data may be compressed and encoded by a standard technique typified by H.264 or HEVC and then transmitted to the end user terminal 190 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 190 without being compressed. In this case, the former performing compression encoding assumes a smartphone or a tablet as the end user terminal 190, and the latter assumes a display capable of displaying an uncompressed image. That is, the image format can be switched according to the type of the end user terminal 190. In addition, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or other transmission methods may be used.

  As described above, the image processing system 100 has three functional domains: a video collection domain, a data storage domain, and a video generation domain. The video collection domain includes a sensor system 110-110z, the data storage domain includes a database 250, a front-end server 230, and a back-end server 270, and the video generation domain includes a virtual camera operation UI 330 and an end user terminal 190.

  Note that the configuration of the image processing system is not necessarily limited to this configuration. For example, the virtual camera operation UI 330 may directly acquire an image from the sensor systems 110a to 110z. However, in the present embodiment, as described above, the configuration is such that the data storage function is arranged in the middle rather than a configuration in which an image is directly acquired from the sensor systems 110a to 110z. Specifically, the front-end server 230 is configured to convert image data, audio data, and meta information thereof generated by the sensor systems 110a to 110z into a common schema and data type of the database 250. Thereby, even if the camera 112 of the sensor systems 110a to 110z is changed to a camera of another model, the changed difference can be absorbed by the front end server 230 and registered in the database 250. That is, this can reduce the possibility that the virtual camera operation UI 330 does not operate properly even when the camera 112 is changed to another model camera.

  Further, the virtual camera operation UI 330 is configured to access the database 250 via the back-end server 270 without directly accessing the database 250. That is, common processing related to image generation processing is executed by the back-end server 270, and application differences related to the operation UI are executed by the virtual camera operation UI 330.

  With such a configuration, in the development of the virtual camera operation UI 330, it is possible to focus on development related to a UI operation device and a UI function request for operating a virtual viewpoint image to be generated. Further, the back-end server 270 can add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 330. In this way, it is possible to flexibly respond to the request for the virtual camera operation UI 330.

  As described above, in the image processing system 100, the virtual viewpoint image is generated by the back-end server 270 based on the image data obtained by photographing the subject from a plurality of directions by the camera 112 (that is, the cameras 112a to 112z). The image processing system 100 according to the present embodiment has been described as a physical configuration in the above description, but the configuration does not necessarily need to be physical, and may be logically configured.

  Next, each node (camera adapter 120, front end server 230, database 250, back end server 270, virtual camera operation UI 330, end user terminal 190) in the image processing system 100 of FIG. 1 will be described in order.

  FIG. 2 is a functional block diagram of the camera adapter 120 in the present embodiment. As illustrated in FIG. 2, the camera adapter 120 includes a network adapter 06110, a transmission unit 06120, an image processing unit 06130, and an external device control unit 06140.

  The network adapter 06110 includes a data transmission / reception unit 06111 and a time control unit 06112. In the present embodiment, a network interface card (NIC) is used as the network adapter 06110. However, the present invention is not limited to the NIC, and other similar interfaces may be used.

  The data transmission / reception unit 06111 performs data communication with the other camera adapter 120, the front-end server 230, the time server 290, and the control station 310 via the daisy chain 170, the network 180b, and the network 310a.

  For example, the data transmission / reception unit 06111 outputs the foreground image and the background image separated by the foreground / background separation unit 06131 from the image captured by the camera 112 to another camera adapter 120. Each camera adapter 120 outputs a foreground image and a background image, thereby generating a virtual viewpoint image based on the foreground image and the background image taken from a plurality of viewpoints. The output destination camera adapter 120 is set according to a predetermined order according to the processing of the data routing processing unit 06122 among the camera adapters 120 of the image processing system 100. Further, regarding the camera adapter 120, there may be a camera adapter 120 that outputs only the foreground image separated from the captured image and does not output the background image.

  The time control unit 06112 is compliant with the Ordinary Clock of the IEEE 1588 standard, for example, and has a function of storing a time stamp of data transmitted to and received from the time server 290 and a function of performing time synchronization with the time server 290. Note that IEEE 1588 has been updated as a standard such as IEEE 1588-2002 and IEEE 1588-2008, and the latter is also called PTPv2 (Precision Time Protocol Version 2). Further, the time synchronization with the time server 290 is not necessarily compliant with the IEEE 1588 standard, and the time synchronization with the time server 290 may be realized by another EtherAVB standard or a unique protocol.

  The transmission unit 06120 controls data transmission to the switching hub 180 and the like via the network adapter 06110. The transmission unit 06120 includes a data compression / decompression unit 06121, a data routing processing unit 06122, a time synchronization control unit 06123, an image / audio transmission processing unit 06124, and a data routing information holding unit 06125. Hereinafter, each function will be described.

  The data compression / decompression unit 06121 compresses data transmitted / received via the data transmission / reception unit 06111 by applying a predetermined compression method, compression rate, and frame rate, and decompresses the compressed data. It has the function to do.

  The data routing processing unit 06122 determines the routing destination of the data received by the data transmitting / receiving unit 06111 and the data processed by the image processing unit 06130, using data held by the data routing information holding unit 06125 described later. The data routing processing unit 06122 further transmits data to the determined routing destination. Then, according to the determination by the data routing processing unit 06122 of each of the plurality of camera adapters 120, the order of the camera adapters 120 that output the foreground image and the background image in the relay format is set in the image processing system 100.

  Note that setting the camera adapter 120 corresponding to the camera 112 focused on the same gazing point as the routing destination has a high correlation between the image frames of the respective cameras 112. Is preferred.

  The time synchronization control unit 06123 has a function of performing time synchronization with the time server 290 based on, for example, the IEEE 1588 standard PTP (Precision Time Protocol). Note that the time synchronization with the time server 290 is not necessarily compliant with the IEEE 1588 standard PTP, and the time synchronization may be realized using another similar protocol.

  The image / audio transmission processing unit 06124 has a function of creating a message for transferring image data or audio data to another camera adapter 120 or the front-end server 230 via the data transmission / reception unit 06111. The message includes image data or audio data and meta information of each data. The meta information includes a time code or sequence number when an image is taken or audio is sampled, a data type, an identifier indicating an individual of the camera 112 or the microphone 111, and the like. In addition, image data or audio data to be transmitted may be compressed by the data compression / decompression unit 06121.

  The image / sound transmission processing unit 06124 receives a message from another camera adapter 120 via the data transmission / reception unit 06111. Then, in accordance with the data type included in the message, the data information fragmented to the packet size defined by the transmission protocol is restored to image data or audio data. If the data is compressed when the data is restored, the data compression / decompression unit 06121 executes decompression processing.

  The data routing information holding unit 06125 holds address information for determining a transmission destination of data transmitted / received by the data transmitting / receiving unit 06111. The routing method will be described later.

  The image processing unit 06130 performs processing on image data captured by the camera 112 and image data received from another camera adapter 120 under the control of the camera control unit 06141. The image processing unit 06130 includes a foreground / background separation unit 06131, a three-dimensional model information generation unit 06132, and a calibration control unit 06133. Hereinafter, each function will be described.

  A foreground / background separator 06131 separates image data captured by the camera 112 into a foreground image and a background image. Specifically, in each of the plurality of camera adapters 120, the foreground / background separation unit 06131 extracts a predetermined area from the captured image of the corresponding camera 112. Here, the predetermined area is, for example, a foreground image acquired as a result of object detection on the captured image, and by this extraction, the foreground / background separation unit 06131 separates the captured image into a foreground image and a background image.

  Here, the object is, for example, a person or the like, and may be a specific person such as a player, a manager, and / or a referee. Further, it may be an object with a predetermined image pattern, such as a ball or a goal. In addition, a moving object may be detected as an object. For example, when a person is set as an important object, a foreground image including a person is separated from a background image not including an object (person) as shown in FIG. 8B, as shown in FIG. To process.

  Thereby, the quality of the image of the part applicable to the object of the virtual viewpoint image produced | generated in the image processing system 100 can be improved. Further, by separating the foreground and the background in each of the plurality of camera adapters 120, the load on the image processing system 100 including the plurality of cameras 112 can be distributed. As a supplement, the predetermined area is not limited to the foreground image, but may be a background image, for example.

  The three-dimensional model information generation unit 06132 uses the foreground image separated by the foreground / background separation unit 06131 and the foreground image received from the other camera adapter 120, for example, by applying the principle of a stereo camera, etc. Generate image information about the original model.

  The calibration control unit 06133 acquires image data necessary for calibration from the camera 112 via the camera control unit 06141, and transmits the image data to the front-end server 230 that executes arithmetic processing related to calibration. In the present embodiment, the calculation process related to calibration is executed by the front-end server 230, but the node that executes the calculation process related to calibration is not necessarily limited to the front-end server 230. Therefore, for example, the arithmetic processing may be executed in another node such as the control station 310 or the camera adapter 120 (in this case, including the other camera adapter 120).

  The calibration control unit 06133 performs calibration during shooting (dynamic calibration) on the image data acquired from the camera 112 via the camera control unit 06141 according to a preset parameter.

  The external device control unit 06140 controls devices connected to the camera adapter 120. The external device control unit 06140 includes a camera control unit 06141, a microphone control unit 06142, a pan head control unit 06143, and a sensor control unit 06144. Hereinafter, each function will be described.

  The camera control unit 06141 is connected to the camera 112 and performs control of the camera 112, acquisition of a captured image, provision of a synchronization signal, time setting, and the like. Here, the control of the camera 112 includes setting and referring to shooting parameters (number of pixels, color depth, frame rate, white balance, etc.), and the state of the camera 112 (shooting, stopping, synchronizing, error, etc.) Acquisition, shooting start and stop, and focus adjustment.

  In this embodiment, the focus adjustment is performed via the camera 112. However, when a detachable lens is attached to the camera 112, the camera adapter 120 is connected to the lens and the lens is directly adjusted. May be performed. Further, the camera adapter 120 may perform lens adjustment such as zooming via the camera 112.

  In addition, the synchronization signal is provided by providing the imaging timing (control clock) to the camera 112 using the time synchronized with the time server 290 by the time synchronization control unit 06123. Furthermore, the time setting is performed by providing the time synchronized with the time server 290 by the time synchronization control unit 06123 using, for example, a time code conforming to the SMPTE12M format. Thereby, the provided time code is added to the image data received from the camera 112. The time code format is not limited to SMPTE12M, and other formats may be used. In addition, the camera control unit 06141 may provide the time code to the image data received from the camera 112 without providing the time code to the camera 112.

  The microphone control unit 06142 is connected to the microphone 111 and performs control of the microphone 111, start and stop of sound collection, acquisition of collected sound data, and the like. The control of the microphone 111 is, for example, gain adjustment, state acquisition, and the like. Similarly to the camera control unit 06141, the microphone control unit 06142 provides audio sampling timing and time code to the microphone 111. As clock information that is the timing of audio sampling, time information from the time server 290 is converted into, for example, a 48 KHz word clock and supplied to the microphone 111.

  The pan head control unit 06143 is connected to the pan head 113 and controls the pan head 113. Control of the camera platform 113 includes, for example, pan / tilt control and status acquisition. The sensor control unit (internal sensor) 06144 is connected to the external sensor 114 and acquires sensor information sensed by the external sensor 114. For example, when a gyro sensor is used as the external sensor 114, information indicating the degree of vibration (vibration information) can be acquired. In this case, for example, using the vibration information acquired by the sensor control unit 06144, the image processing unit 06130 can generate an image in which vibration is suppressed prior to processing by the foreground / background separation unit 06131.

  For example, the vibration information is used when image data of an 8K camera is cut out with a size smaller than the original 8K size in consideration of the vibration information and is aligned with the image of the adjacent camera 112. Thereby, even if it is a case where the frame vibration of a building propagates to each camera at a different frequency, alignment can be performed. As a result, it is possible to generate electronically image-stabilized image data (that is, image data in which the influence of vibration is reduced by image processing), and the number of cameras 112 in the image computing server 200 The combined processing load can be reduced. Note that the sensor of the sensor system 110 is not necessarily limited to the external sensor 114, and the same effect can be obtained even if it is a sensor built in the camera adapter 120.

  FIG. 3 is a functional block diagram of the image processing unit 06130 inside the camera adapter 120. Hereinafter, a supplementary explanation will be given for each of the calibration control unit 06133, the foreground / background separation unit 06131, and the three-dimensional model information generation unit 06132.

  The calibration control unit 06133 stabilizes the position of the image by reducing color blurring caused by camera vibration and color correction processing for suppressing color variation for each camera with respect to the input image. Blur correction processing (electronic image stabilization processing) is performed.

  As shown in FIG. 3, the foreground / background separation unit 06131 includes a foreground separation unit 05001, a background update unit 05003, and a background cutout unit 05004. The foreground separation unit 05001 performs a foreground image separation process on the image data that has been aligned with respect to the image of the camera 112 by comparison with the background image 05002. The background update unit 05003 generates a new background image using the image in which the background image 05002 and the camera 112 are aligned, and updates the background image 05002 to a new background image. The background cutout unit 05004 performs control to cut out a part of the background image 05002.

  As shown in FIG. 3, the 3D model information generation unit 06132 includes a 3D model processing unit 05005, another camera foreground reception unit 05006, and a camera parameter reception unit 05007. The three-dimensional model processing unit 05005 uses the foreground image separated by the foreground separation unit 05001 and the foreground image of the other camera 112 received via the transmission unit 06120, for example, based on the principle of a stereo camera. The image information regarding is sequentially generated. The other camera foreground receiving unit 05006 receives the foreground image obtained by separating the foreground and background with the other camera adapter 120. The camera parameter receiving unit 05007 receives camera-specific internal parameters (for example, focal length, image center, lens distortion parameters, etc.) and external parameters (for example, rotation matrix, position vector, etc.) representing the position and orientation of the camera. . These parameters are information acquired by a calibration process described later, and are transmitted and set from the control station 310 to the target camera adapter 120.

  FIG. 4 is a functional block diagram of the front-end server 230 in the present embodiment. As shown in FIG. 4, the front-end server 230 includes various functional blocks (control unit 02110-DB access control unit 02190). Hereinafter, each function will be described.

  The control unit 02110 includes a CPU, a DRAM, a storage medium such as an HDD or NAND memory that stores program data and various data, and hardware such as Ethernet (registered trademark). DRAM is an abbreviation for Dynamic Random Access Memory.

  The control unit 02110 controls each functional block of the front end server 230 and the entire system of the front end server 230. In addition, the control unit 02110 switches operation modes such as a calibration operation, a preparatory operation before photographing, and an operation during photographing by performing mode control. The control unit 02110 further receives a control instruction from the control station 310 through Ethernet (registered trademark), and performs switching of each mode, input / output of data, and the like. Similarly, the control unit 02110 acquires stadium CAD data (stadium shape data) from the control station 310 through the network, and transmits the stadium CAD data to the CAD data storage unit 02135 and the imaging data file generation unit 02180. The stadium CAD data (stadium shape data) in the present embodiment is three-dimensional data indicating the shape of the stadium, and may be data representing a mesh model or other three-dimensional shapes, and is not necessarily limited to the CAD format.

  The data input control unit 02120 is connected to the camera adapter 120 via a communication path such as Ethernet (registered trademark) and the switching hub 180 via a network. The data input control unit 02120 acquires a foreground image, a background image, a three-dimensional model (three-dimensional shape) of the subject, audio data, and camera calibration photographed image data from the camera adapter 120 via the network.

  Here, the foreground image is image data based on the foreground area of the captured image for generating the virtual viewpoint image, and the background image is image data based on the background area of the captured image. The camera adapter 120 identifies the foreground area and the background area according to the result of detection processing of a predetermined object for the captured image by the camera 112, and generates the foreground image and the background image. The predetermined object is, for example, a person or the like, and may be a specific person such as a player, a manager, and / or a referee. The predetermined object may include an object having a predetermined image pattern such as a ball or a goal. In addition, a moving object may be detected as the predetermined object.

  The data input control unit 02120 performs received data compression / decompression, data routing processing, and the like. The data input control unit 02120 further transmits the foreground image and the background image to the data synchronization unit 02130, and transmits camera calibration photographed image data to the calibration unit 02140. Note that both the control unit 02110 and the data input control unit 02120 have a communication function via a network such as Ethernet (registered trademark), but even if the control unit 02110 and the data input control unit 02120 communicate with each other using the communication function. Good. In this case, for example, when receiving an instruction by a control command from the control station 310 or stadium CAD data, the data input control unit 02120 transmits such data to the control unit 02110.

  The data synchronization unit 02130 temporarily stores (buffers) the data acquired from the camera adapter 120 in the internal DRAM until the foreground image, background image, audio data, and 3D model data are prepared. This is because the reception order of network packets is not guaranteed for the data transferred from each camera adapter 120 by the network, and it is necessary to prepare data necessary for file generation. Note that the data aligned here is data necessary to generate a file in the shooting data file generation unit 02180 described later. Further, hereinafter, the foreground image, the background image, the sound data, and the 3D model data are collectively referred to as shooting data.

  Meta information such as routing information, time code information (time information), and a camera identifier is added to the shooting data, and the data synchronization unit 02130 confirms the attribute of the data based on the meta information. As a result, the data synchronization unit 02130 determines that the data is at the same time, and confirms that the data is ready.

  When the data is ready, the data synchronization unit 02130 sends the foreground image and the background image to the image processing unit 02150, the 3D model data to the 3D model combination unit 02160, and the audio data to the shooting data file generation unit 02180, respectively. Send. Note that the background image may be taken at a frame rate different from that of the foreground image. In this case, for example, when the frame rate of the background image is 1 fps, one background image is acquired every second, and therefore, for the time when the background image is not acquired, all the data is obtained without the background image. It's okay.

  In addition, the data synchronization unit 02130 notifies the database 250 of information indicating that data cannot be collected when the data is not ready after a predetermined time has elapsed. For example, the database 250 at the subsequent stage stores information indicating data loss together with the camera number and frame number when storing data. Thus, according to the viewpoint instruction from the virtual camera operation UI 330 to the back-end server 270, whether or not a desired image can be formed from the captured images of the camera 112 collected data can be automatically notified before rendering. . As a result, the visual load on the operator of the virtual camera operation UI 330 can be reduced.

  The CAD data storage unit 02135 stores the three-dimensional data indicating the stadium shape received from the control unit 02110 in a storage medium such as a DRAM or HDD or NAND memory. When the CAD data storage unit 02135 receives a request for stadium shape data, the CAD data storage unit 02135 transmits the stored stadium shape data to the image combining unit 02170.

  The calibration unit 02140 performs a camera calibration operation, and transmits camera parameters acquired by the calibration to a non-photographed data file generation unit 02185 described later. At the same time, the calibration unit 02140 holds camera parameters in its own storage area, and provides camera parameter information to a 3D model combining unit 02160 described later.

  The image processing unit 02150 matches colors and luminance values between cameras with respect to the foreground image and the background image, develops processing when RAW image data is input, and error compensation at the time of image decoding and camera lens Processing such as distortion correction is executed. Then, the foreground image subjected to the image processing is transmitted to the shooting data file generation unit 02180, and the background image is transmitted to the image combination unit 02170.

  The 3D model combining unit 02160 combines the 3D model data at the same time acquired from the camera adapter 120 using the camera parameters generated by the calibration unit 02140. Then, the 3D model combination unit 02160 generates 3D model data of the foreground image in the entire stadium using a method called VisualHull. The generated three-dimensional model data is transmitted to the shooting data file generation unit 02180.

  When the image combining unit 02170 obtains the background image from the image processing unit 02150 and obtains the three-dimensional shape data of the stadium from the CAD data storage unit 02135, the image combining unit 02170 obtains the background image with respect to the coordinates of the obtained three-dimensional shape data of the stadium. Identify the location. When the position of the stadium three-dimensional shape data (stadium shape data) is specified for each of the background images, the image combining unit 02170 combines the background images into one background image. Note that the back-end server 270 may execute the generation of the three-dimensional shape data of the background image.

  The shooting data file generation unit 02180 is combined with the audio data from the data synchronization unit 02130, the foreground image from the image processing unit 02150, the 3D model data from the 3D model combining unit 02160, and the 3D shape from the image combining unit 02170. Get the background image. Then, the captured data file generation unit 02180 outputs the acquired data to the DB access control unit 02190.

  Here, the imaging data file generation unit 02180 outputs these data in association with each other based on each time information. However, in this case, some of these data may be output in association with each other. For example, the shooting data file generation unit 02180 outputs the foreground image and the background image in association with each other based on the time information of the foreground image and the time information of the background image. In addition, for example, the shooting data file generation unit 02180 handles foreground images, background images, and 3D model data based on foreground image time information, background image time information, and 3D model data time information. Output. Note that the shooting data file generation unit 02180 may output the data associated with each other in the form of a file for each data type, or may output a plurality of types of data collectively for each time indicated by the time information. it can.

  Then, the image data thus associated is output to the database 250 by the DB access control unit 02190 described later, so that the back-end server 270 can determine the virtual viewpoint from the foreground image and the background image corresponding to the time information. An image can be generated.

  When the foreground image acquired by the data input control unit 02120 and the background image have different frame rates, it is difficult for the shooting data file generation unit 02180 to always output the foreground image and the background image at the same time in association with each other. is there. Therefore, the shooting data file generation unit 02180 associates and outputs the time information of the foreground image and the background image having the time information having a relationship based on a predetermined rule. Here, the background image having the time information of the foreground image and the time information having a relationship based on a predetermined rule is, for example, the closest to the time information of the foreground image among the background images acquired by the shooting data file generation unit 02180. It is a background image having time information. In this way, by associating the foreground image and the background image based on a predetermined rule, even if the foreground image and the background image have different frame rates, the virtual viewpoint image can be obtained from the foreground image and the background image captured at a close time. Can be generated.

  As a supplement, the method for associating the foreground image with the background image is not necessarily limited to the above method. For example, among the background images acquired at a time before the foreground image as a background image having time information having a relationship based on a predetermined rule with the time information of the foreground image, the foreground A background image having time information closest to the time information of the image may be set. According to this method, the associated foreground image and background image can be output with low delay without waiting for acquisition of a background image having a lower frame rate than the foreground image. In addition, among the background images acquired at a time after the foreground image as a background image having time information having a relationship based on a predetermined rule with the time information of the foreground image, the foreground A background image having time information closest to the time information of the image may be set.

  When the non-photographed data file generation unit 02185 acquires the camera parameters from the calibration unit 02140 and the three-dimensional shape data of the stadium from the control unit 02110, the non-photographing data file generation unit 02185 shapes the data according to the file format. Then, the formed data is transmitted to the DB access control unit 02190. The camera parameters and stadium shape data input to the non-photographing data file generation unit 02185 are individually formed according to the file format. That is, when any one of the data is received, the non-photographed data file generation unit 02185 individually transmits the data to the DB access control unit 02190.

  The DB access control unit 02190 is connected to the database 250 so that high-speed communication is possible using InfiniBand or the like. The DB access control unit 02190 transmits the files received from the imaging data file generation unit 02180 and the non-imaging data file generation unit 02185 to the database 250. As described above, in the present embodiment, the shooting data associated with the shooting data file generation unit 02180 based on the time information is output to the database 250 connected to the front-end server 230 via the network.

  Note that the output destination of the shooting data associated based on the time information by the shooting data file generation unit 02180 is not necessarily limited to this. Therefore, for example, the front-end server 230 may output the shooting data associated based on the time information to the back-end server 270 that is connected to the front-end server 230 via the network and generates a virtual viewpoint image. Good. Further, the front end server 230 may output shooting data associated based on the time information to both the database 250 and the back end server 270.

  In addition, in the present embodiment, the front-end server 230 is described as a specification for associating the foreground image with the background image. However, the present invention is not necessarily limited thereto, and the database 250 can also perform the association. For example, the database 250 acquires a foreground image and a background image having time information from the front-end server 230, associates the foreground image and the background image based on each time information, and stores them in a storage area included in the database 250 itself. It can also be output.

  FIG. 5 is a functional block diagram of the data input control unit 02120 inside the front-end server 230. As shown in FIG. 5, the data input control unit 02120 includes a server network adapter 06210, a server transmission unit 06220, and a server image processing unit 06230. Hereinafter, a supplementary explanation will be given for each of the server network adapter 06210, the server transmission unit 06220, and the server image processing unit 06230.

  The server network adapter 06210 has a server data receiving unit 06221 and receives data transmitted from the camera adapter 120. The server transmission unit 06220 performs processing on the data received from the server data reception unit 06221. Hereinafter, the server data decompression unit 06221, the server data routing processing unit 06222, the server image / audio transmission processing unit 06223, and the server data routing information holding unit 06224 will be supplementarily described.

  The server data decompression unit 06221 decompresses the compressed data. The server data routing processing unit 06222 determines a data transfer destination based on routing information such as an address held by a server data routing information holding unit 06224, which will be described later, and transfers the data received from the server data receiving unit 06221.

  The server image / audio transmission processing unit 06223 receives a message from the camera adapter 120 via the server data reception unit 06221, and converts the fragmented data into image data or audio data according to the data type included in the message. Restore. If the restored image data or audio data is compressed, the server data decompression unit 06221 performs decompression processing.

  The server data routing information holding unit 06224 holds address information for determining a transmission destination of data received by the server data receiving unit 06221. The routing method will be described later.

  The server image processing unit 06230 executes processing related to image data or audio data received from the camera adapter 120. The processing content includes, for example, camera number, image frame shooting time, image size, image format, and image coordinate attribute information according to the data entity of the image data (foreground image, background image, and 3D model information). For example, shaping processing into a format to which etc. are assigned.

  FIG. 6 is a functional block diagram of the database 250. The control unit 02410 includes a CPU, a DRAM, a storage medium such as an HDD or a NAND memory that stores program data and various data, and hardware such as Ethernet (registered trademark). The control unit 02410 controls each functional block of the database 250 and the entire system of the database 250.

  The data input unit 02420 receives shooting data and non-shooting data files from the front-end server 230 through high-speed communication such as InfiniBand. The received file is transmitted to the cache (RAM) 02440. In addition, the meta information of the received shooting data is read, and a database table is created so that the acquired data can be accessed based on information such as time code information, routing information, and camera identifier recorded in the meta information. To do.

  The data output unit 02430 determines whether the data requested from the back-end server 270 is stored in a cache 02440, a primary storage 02450, or a secondary storage 02460, which will be described later. The data output unit 02430 reads out data from the storage destination and transmits it to the back-end server 270 by high-speed communication such as InfiniBand.

  The cache 02440 includes a storage device such as a DRAM capable of realizing high-speed input / output throughput, and stores the shooting data and non-shooting data acquired from the data input unit 02420 in the storage device. A certain amount of stored data is retained, and when data exceeding that amount is input, old data is written to the primary storage 02450 as needed, and written data is overwritten by new data.

  Note that the data stored in a certain amount in the cache 02440 is photographing data for at least one frame. Thereby, when the image rendering process is executed in the back-end server 270, the throughput in the database 250 can be minimized, and the latest image frame can be rendered continuously with low delay. Here, in order to achieve the above object, the cached data needs to include a background image. For this reason, when shooting data of a frame that does not have a background image is cached, the background image on the cache is not updated and is held on the cache as it is.

  The capacity of the DRAM that can be cached is set by a cache frame size preset in the system or by an instruction from the control station. Note that the non-photographed data is copied to the primary storage immediately because the frequency of input / output is low and high-speed throughput is not required before a game or the like. Also, the cached data is read by the data output unit 02430.

  The primary storage 02450 is configured by connecting storage media such as SSDs in parallel. In the primary storage 02450, writing processing of a large amount of data from the data input unit 02420 and data reading processing from the data output unit 02430 are realized at the same time, and the processing speed is increased. In the primary storage 02450, data stored in the cache 02440 is written in order from the oldest data.

  The secondary storage 02460 is composed of an HDD, a tape medium, and the like. The secondary storage 02460 is a medium that emphasizes a large capacity rather than high speed, and is inexpensive and suitable for long-term storage compared to the primary storage. Note that the data stored in the primary storage 02450 is written to the secondary storage 02460 as a backup of the data after the photographing is completed.

  FIG. 7 is a functional block diagram of the back-end server 270. As illustrated in FIG. 7, the back-end server 270 includes various functional blocks (data reception unit 03001-rendering mode management unit 03014). Hereinafter, each function will be described.

  The data receiving unit 03001 receives data transmitted from the database 250 and the controller 300. The data receiving unit 03001 receives from the database 250 three-dimensional data (stadium shape data) indicating the shape of the stadium, foreground image, background image, three-dimensional model of the foreground image (hereinafter referred to as the foreground three-dimensional model), and sound. Receive. Further, the data receiving unit 03001 receives virtual camera parameters from the controller 300 that designates a viewpoint related to generation of a virtual viewpoint image. Here, the virtual camera parameters are data indicating the position and orientation of the virtual viewpoint, and for example, an external parameter matrix and an internal parameter matrix are used as the virtual camera parameters.

  Note that the data received by the data receiving unit 03001 from the controller 300 is not necessarily limited to virtual camera parameters. Information acquired from the controller 300 may include at least one of a viewpoint designation method, information for specifying an application that the controller is operating, identification information for the controller 300, and identification information for a user who uses the controller 300. Good.

  Further, the data receiving unit 03001 may acquire the same information as the above-described information output from the controller 300 from the end user terminal 190. Further, the data receiving unit 03001 may acquire information related to the plurality of cameras 112 from external devices such as the database 250 and the controller 300. Here, the information regarding the plurality of cameras 112 is, for example, information regarding the number of the plurality of cameras 112, information regarding the operation state of the plurality of cameras 112, and the like. Note that the operation state of the camera 112 includes, for example, at least one of a normal state, a failure state, a standby state, a start state, and a restart state of the camera 112.

  The background texture pasting unit 03002 obtains the background image from the background mesh model management unit 03013 and pastes the background image as a texture to the three-dimensional space shape indicated by the background mesh model (stadium shape data). As a result, the background texture pasting unit 03002 generates a textured background mesh model. Here, the mesh model is data representing a three-dimensional space shape as a set of surfaces, such as CAD data. A texture is an image that is pasted on a three-dimensional space shape in order to express the surface texture of an object.

  The foreground texture determination unit 03003 determines the texture information of the foreground 3D model from the foreground image and the foreground 3D model group. The foreground texture boundary color matching unit 03004 executes color matching of the texture boundary from the texture information of the foreground 3D model and the 3D model group, and generates a colored foreground 3D model group for each foreground object. The virtual viewpoint foreground image generation unit 03005 performs perspective transformation so that the foreground image group looks from the virtual viewpoint based on the virtual camera parameters.

  The rendering unit 03006 renders the background image and the foreground image based on the generation method used for generating the virtual viewpoint image determined by the rendering mode management unit 03014, and generates a virtual viewpoint image of the entire view. Here, two rendering modes of model-based rendering (Model-Based Rendering: MBR) and image-based rendering (Image-Based Rendering: IBR) are used as the virtual viewpoint image generation method.

  MBR is a method of generating a virtual viewpoint image using a three-dimensional model generated based on a plurality of captured images when the subject is captured from a plurality of directions. Specifically, using the three-dimensional shape (model) of the target scene acquired by a three-dimensional shape restoration method such as a visual volume intersection method or Multi-View-Stereo (MVS), the appearance of the scene from the virtual viewpoint is determined. This is a technique for generating images. In the present embodiment, when MBR is used as the rendering mode, a foreground model is generated by combining the background mesh model and the foreground 3D model group generated by the foreground texture boundary color matching unit 03004. A virtual viewpoint image is generated.

  IBR is a technique for generating a virtual viewpoint image that reproduces the appearance from a virtual viewpoint by transforming and synthesizing an input image group obtained by photographing a target scene from a plurality of viewpoints. In this embodiment, when IBR is used as a rendering mode, a background image viewed from a virtual viewpoint is generated based on a background texture model, and a foreground image generated by the virtual viewpoint foreground image generation unit 03005 is synthesized there. A virtual viewpoint image is generated. When IBR is used as a rendering mode, a virtual viewpoint image is generated based on fewer photographed images than photographed images used when rendering by MBR (that is, photographed images used for generating a three-dimensional model). can do. In addition, as a supplement, the rendering unit 03006 can use rendering methods other than the above-described MBR and IBR.

  The rendering mode management unit 03014 determines a generation method (that is, rendering mode) used for generating the virtual viewpoint image, and holds the determined result. In the present embodiment, the rendering mode management unit 03014 determines a rendering mode used for rendering from a plurality of rendering modes.

  The rendering mode is determined based on information acquired by the data receiving unit 03001. For example, when the number of cameras specified from the acquired information is equal to or less than a threshold value, the rendering mode management unit 03014 determines the generation method used for generating the virtual viewpoint image to be IBR. As described above, when the number of cameras is small, by using IBR, it is possible to avoid a decrease in image quality of the virtual viewpoint image due to a decrease in accuracy of the three-dimensional model when MBR is used.

  On the other hand, when the number of cameras specified from the acquired information is greater than the threshold, the rendering mode management unit 03014 determines the generation method used for generating the virtual viewpoint image to be MBR. As described above, when the number of cameras is large, the specifiable range of viewpoints can be expanded by generating a virtual viewpoint image using MBR.

  Note that the rendering mode management unit 03014 may determine the generation method (that is, the rendering mode) based on, for example, the allowable processing delay time from shooting to image output. At this time, MBR is used when priority is given to the degree of freedom of view even if the delay time is long, and IBR is used when a short delay time is required.

  For example, when the data reception unit 03001 acquires information indicating that the controller 300 or the end user terminal 190 can specify the height of the viewpoint, the rendering mode management unit 03014 is used to generate a virtual viewpoint image. The generation method is determined as MBR. As a result, it is possible to prevent the request for changing the height of the viewpoint by the user from being accepted due to the generation method being IBR.

  Thus, by determining the generation method of the virtual viewpoint image according to the situation, the virtual viewpoint image can be generated by an appropriate generation method. In addition, by adopting a configuration in which a plurality of rendering modes can be switched as required (that is, by flexibly configuring the system), the present invention can be easily applied to subjects other than the stadium.

  Note that the rendering mode held by the rendering mode management unit 03014 may be a system preset in the system. Further, a user who operates the virtual camera operation UI 330 or the end user terminal 190 may arbitrarily set the rendering mode. In addition, the virtual viewpoint image generation method determined by the rendering mode management unit 03014 is not necessarily limited to the rendering method, and the rendering mode management unit 03014 may determine a processing method other than rendering for generating the virtual viewpoint image. it can.

  The virtual viewpoint sound generation unit 03007 generates sound (sound group) that can be heard at the virtual viewpoint based on the virtual camera parameters. The synthesizing unit 03008 synthesizes the image group generated by the rendering unit 03006 and the audio generated by the virtual viewpoint audio generation unit 03007 to generate virtual viewpoint content.

  The image output unit 0309 outputs virtual viewpoint content to the controller 300 and the end user terminal 190 using Ethernet (registered trademark). Note that the transmission means to the outside is not limited to Ethernet (registered trademark), and signal transmission means such as SDI, DisplayPort, and HDMI (registered trademark) may be used. Further, the back-end server 270 may output the virtual viewpoint image generated by the rendering unit 03006 (that is, a virtual viewpoint image that does not include sound).

  The foreground object determination unit 03010 determines a foreground object group to be displayed from the virtual camera parameters and position information of the foreground object indicating the position of the foreground object included in the foreground three-dimensional model, and outputs a foreground object list To do. That is, the foreground object determination unit 03010 executes a process of mapping the virtual viewpoint image information to the physical camera 112.

  In the present embodiment, mapping processing is executed in accordance with the rendering mode determined by the rendering mode management unit 03014. For this reason, a control unit that determines a plurality of foreground objects is mounted in the foreground object determination unit 03010 and performs control in conjunction with the rendering mode.

  The request list generation unit 03011 generates a request list for requesting the database 250 for the foreground image group and the foreground 3D model group corresponding to the foreground object list at the specified time, and the background image and audio data. For the foreground image group and the foreground three-dimensional model group, data selected in consideration of the virtual viewpoint is required of the database 250. However, for the background image and the audio data, all data related to the frame is stored in the database 250. As required. After the back-end server 270 is started, a background mesh model request list is generated until the background mesh model is acquired.

  The request data output unit 03012 outputs a data request command to the database 250 based on the input request list. The background mesh model management unit 03013 stores the background mesh model received from the database 250.

  In the present embodiment, the back-end server 270 is described as a specification for executing both the determination of the generation method of the virtual viewpoint image and the generation of the virtual viewpoint image, but the present invention is not necessarily limited thereto. Therefore, the back-end server 270 may generate the virtual viewpoint image based on the input information indicating the generation method without determining the virtual viewpoint image generation method. In that case, for example, the generation method used by the front-end server 230 to generate a virtual viewpoint image based on information about a plurality of cameras 112, information output from a device that specifies a viewpoint for generating a virtual viewpoint image, or the like. To decide. The front-end server 230 outputs image data based on the image taken by the camera 112 to the database 250 and information indicating the determined generation method to the back-end server 270. Then, the back-end server 270 generates a virtual viewpoint image based on the information indicating the generation method output from the front-end server 230.

  As described above, when the front-end server 230 determines the generation method, the processing load caused by the database 250 and the back-end server 270 processing data for image generation using a method different from the determined method. Can be reduced. As a supplement, when the back-end server 270 determines the generation method as in the present embodiment, the database 250 holds data that can correspond to a plurality of generation methods, and thus a plurality of generation methods corresponding to each of the plurality of generation methods. A virtual viewpoint image can be generated. As described above, by configuring the system, a desired virtual viewpoint image can be generated from captured images acquired by a plurality of cameras.

  Next, the code amount control process in the camera adapter 120 will be described. This code amount control process is executed in order to appropriately determine the encoding process for each viewpoint so that the code amount conforms to the transmission path in the image processing system.

  Hereinafter, the procedure of the code amount control process will be described with reference to the flowchart of FIG. In the following description, the term target code amount is used to mean a code amount set in each camera in order to transmit data generated in the image processing system in real time. Specifically, in the present embodiment, the target code amount is a value obtained by dividing a usable band by the number of viewpoints (the number of cameras) in order to transmit video in the communication band of the transmission path. That is, the target code amount serving as an index of image quality can be set based on the transmission capacity of the transmission path.

  In step S901, the camera adapter 120 collates the visual field overlap map, and acquires the number of cameras capturing itself for each foreground. Here, the field-of-view overlap map is a map indicating how much the fields of view of other cameras overlap in the field of view of the camera being processed (that is, the degree of overlap).

  Hereinafter, the visual field overlap map will be described with reference to FIG. FIG. 10A is a diagram for explaining a field-of-view overlap map. In FIG. 10A, four cameras are arranged, and the field of view of each camera is schematically shown. For ease of explanation, the number of cameras is four, but in reality, using more cameras than this is desirable from the viewpoint of the image quality of the virtual viewpoint.

  As shown in FIG. 10A, on the stadium field, the number of cameras in the field of view (hereinafter referred to as field overlap) differs depending on the area. A visual field overlap map is obtained by mapping the degree of visual field overlap based on the visual field of each camera. As described above, the visual field overlap map is created (acquired) based on the relative positional relationship between the three-dimensional shape of the stadium and the camera arrangement. In the present embodiment, it is assumed that a visual field overlap map is created as advance data and is distributed in advance from the control station 310 to each target camera adapter.

  FIG. 10B is a visual field overlap map in Cam3 (camera 3) of FIG. 10A, and the numbers in the map indicate the overlapping degree of the regions. Here, for example, as shown in FIG. 10C, when the foreground A exists, the degree of overlap in the foreground A is 2, and the degree of overlap in the foregrounds B and C is 4.

  In this way, it is possible to determine how many cameras the foreground is captured by creating a field-of-view overlap map in the field of view of each camera and determining where the foreground is located on the map. In other words, in this case, the camera adapter 120 executes a process for deriving the number of cameras that shoot an object as a foreground (camera number deriving process). It is desirable to use the coordinates of the foreground foot as the foreground coordinates used for this determination. In that case, for example, the midpoint of the lower side of the circumscribed rectangle of the foreground may be used. However, if it is predicted that occlusion will occur at the foot, the coordinates of the foot are calculated based on the size of the subject (for example, 170 cm for a person) from the midpoint of the upper side of the circumscribed rectangle. May be.

  In addition, when determining the coordinates of the foreground, image information regarding the three-dimensional model generated by the three-dimensional model information generation unit 06132 may be used. As described above, in the process of generating the image information related to the three-dimensional model, for example, the principle of a stereo camera is applied (that is, the distance information from the foreground to the camera 112 is processed), the coordinates of the foreground Can be determined (derived). Further, when the foreground extends over a plurality of areas in the field-of-view overlap map, the smallest degree of overlap within that area is used.

  In step S902, the camera adapter 120 determines whether or not the frame of interest as a processing target is the first frame (that is, the initial frame) from which shooting has started. If the camera adapter 120 determines that the frame of interest as the processing target is the first frame for which shooting has started (S902 Yes), the process proceeds to step S903. If it is determined that the frame of interest as a processing target is not the first frame for which shooting has started (S902: No), the process proceeds to step S905.

  When the process proceeds to step S903, the camera adapter 120 acquires (measures) the number of foreground pixels reflected in the connected camera 112 for each field overlap degree. Specifically, for example, in FIG. 10C, the number of pixels with field overlap 1 is the number of pixels in foreground E, the number of pixels with field overlap 2 is the number of pixels in foreground A, and the number of pixels with field overlap 3 is As for the number of pixels of the foreground D and the number of pixels of the field overlap degree 4, the total value of the number of pixels of the foreground B and the number of pixels of the foreground C is measured.

  When the camera adapter 120 acquires the number of foreground pixels, in step S904, the camera adapter 120 determines a quantization parameter to be applied to each foreground during encoding. It should be noted that the camera adapter 120, when executing the processing of step S904, performs the quantization parameter / compression ratio correspondence table shown in FIG. 11 (a) and the visual field overlap / quantization parameter shown in FIG. 11 (b). A set correspondence table is used. Hereinafter, the contents will be supplemented with reference to FIG.

  First, the quantization parameter / compression ratio correspondence table will be described. This quantization parameter / compression rate correspondence table is a table indicating the compression rate corresponding to the quantization parameter, and is a table for predicting whether or not a desired code amount is obtained when the quantization parameter is set. The quantization parameter / compression ratio correspondence table is created based on results obtained in advance through simulations, preliminary experiments, or the like.

  FIG. 11A is an example of a quantization parameter / compression ratio correspondence table. In FIG. 11A, qp is a quantization parameter, and br is a compression rate. As shown in FIG. 11A, for example, when the quantization parameter is 100 (qp = 100), the compression rate is 70% (br = 0.70), and the quantization parameter is 50 (qp = 50). In this case, the compression rate is 35% (br = 0.35). Hereinafter, the compression rate corresponding to the quantization parameter qp is referred to as br [qp].

  Next, the field overlap / quantization parameter set correspondence table will be described. This view overlap / quantization parameter set correspondence table is used to control the amount of code generated in the camera being processed. FIG. 11B is an example of a table of visual field overlap / quantization parameter set correspondence. dn is a visual field overlap degree, and ql is a stage of a quantization parameter set (hereinafter referred to as a quantization level). As shown in FIG. 11B, for example, when the quantization level is set to 2 (ql = 2), the quantization parameter qp with a field overlap degree of 10 (dn = 10) is 60, and the field overlap is also shown. The quantization parameter qp with a degree of 1 (dn = 1) is 100. As described above, the quantization parameter is set so as to be inversely correlated with the visual field overlap degree. That is, the image quality is set so that the foreground image has a high image quality from the foreground image with a small number of cameras capturing the foreground.

  Note that this field-of-view overlap / quantization parameter set correspondence table may be created while confirming the image quality of the virtual viewpoint content through simulation or preliminary experiment based on the system policy. Further, hereinafter, the quantization parameter corresponding to the visual field overlap degree dn and the quantization level ql is described as qp [dn, ql].

  In the present embodiment, the foreground quantization parameter is determined by searching for the minimum quantization level that achieves the target code amount. The processing for determining each foreground quantization parameter will be supplemented by using FIG.

  In step S1201, the camera adapter 120 sets the current quantization level curQL to 0 (curQL = 0). Next, in step S1202, the camera adapter 120 acquires a quantization parameter (quantization parameter set) for each visual field overlap degree associated with the current quantization level curQL.

  In step S1203, the camera adapter 120 calculates the predicted code amount based on the following expression from the number of foreground pixels for each field overlap obtained in step S903 in FIG. 9 and the current quantization level. In the following equation, the number of cameras is N, and the number of foreground pixels corresponding to the visual field overlap is fpix [dn].

  In step S1204, the camera adapter 120 determines whether or not the predicted code amount calculated in step S1203 is smaller than the target code amount. The camera adapter 120 shifts the process to step S1205 when the predicted code amount calculated in step S1203 is smaller than the target code amount, and performs the process when the predicted code amount calculated in step S1203 is equal to or larger than the target code amount. The process proceeds to step S1206.

  When the process proceeds to step S1205, the camera adapter 120 assigns a quantization parameter corresponding to the quantization level curQL to each foreground according to the degree of visual field overlap. When the process proceeds to step S1206, the camera adapter 120 counts up the current quantization level curQL. By executing the above processing, the quantization parameter for each foreground can be determined.

  Here, returning to the flowchart of FIG. 9 again, as described above, when it is determined that the frame of interest as the processing target is not the first frame from which shooting was started (No in S902), the process proceeds to step S905. . In step S905, the camera adapter 120 determines the quantization parameter for each foreground based on the quantization level determined after the processing of the previous frame in the camera being processed.

  In step S <b> 906, the camera adapter 120 executes an encoding process (that is, generates encoded data) using the quantization parameter for each foreground acquired up to the previous processing step. The encoding processing algorithm in step S906 may be any algorithm that can execute quantization based on the quantization parameter. For example, JPEG or JPEG2000 is used.

  In step S907, the camera adapter 120 calculates a difference ΔS between the target code amount and the actually output code amount (hereinafter, actual code amount). That is, the difference ΔS is calculated based on the following equation.

  In step S908, the camera adapter 120 determines whether or not the absolute value of the difference ΔS is greater than the threshold value T. If it is determined that the absolute value of the difference ΔS is greater than the threshold T (Yes in S908), the process proceeds to step S909. If it is determined that the absolute value of the difference ΔS is equal to or less than the threshold T (No in S908), the process of FIG. finish.

  In step S909, the camera adapter 120 updates the quantization level. The processing in step S909 is executed in order to absorb the difference between the actual code amount and the target code amount in the next frame. Specifically, when the difference ΔS is larger than the threshold T, that is, when the actual code amount exceeds the target code amount by a predetermined width T or more, the quantization level is counted up (+1 is added). As a result, the code amount can be adjusted to decrease in the next frame. When the difference ΔS is smaller than the threshold −T, that is, when the actual code amount is less than or equal to the target code amount by a predetermined width T, the quantization level is counted down (-1). As a result, the code amount can be adjusted to increase in the next frame.

  However, the process of updating the quantization level is executed within a predetermined range (0-6 in FIG. 11) while limiting so as not to exceed the quantization level. It should be noted that the threshold value T is set to be large when the buffer provided in the camera adapter 120 is relatively large, and is set to be small otherwise. When the camera adapter 120 updates the quantization level, the process of FIG. 9 ends.

  By processing as described above, it is possible to control the amount of code while suppressing image quality deterioration even though the degree of visual field overlap is small based on the processing result in the previous frame, so that large image quality deterioration does not occur in the virtual viewpoint content. Real-time transmission can be realized.

(Second Embodiment)
In the first embodiment described above, in order to speed up the processing, as shown in FIG. 9, the quantization level in the frame to be processed is determined based on the quantization level determined in advance in the processing of the previous frame. did.

  However, with respect to the quantization level, the prediction code amount may be calculated for each frame based on the number of foreground pixels for each field overlap, and the quantization parameter may be determined based on the prediction code amount. With this configuration, it is possible to determine the quantization level based on the contents of the frame being processed (that is, the number of foreground pixels for each visual field overlap), so that the actual code amount can be made closer to the target code amount. it can.

(Third embodiment)
In the first embodiment described above, a target code amount is set in advance for each camera, and the following is performed based on whether or not the absolute value of the difference ΔS between the target code amount and the actual code amount is larger than the threshold T. The quantization level (that is, the encoding parameter) of each frame is set. However, in the first embodiment, when the actual code amount exceeds the target code amount in many cameras on the transmission line, it is assumed that transmission is delayed.

  Therefore, in this embodiment, when the camera being processed is the v-th camera, the camera being processed depends on the content output by the camera adapter 120 upstream (i.e., 1 to v-1). Control encoding parameters. Hereinafter, the code amount control process according to the present embodiment will be described with reference to FIG. The entire system configuration and processes other than the code amount control process are the same as those in the first embodiment, and the code amount control process is performed for each frame on the premise of the second embodiment described above. Quantization parameters are determined based on the number of foreground pixels for each field overlap.

  First, in step 1301, the camera adapter 120 acquires the difference ΔUS in the accumulated target code amount transmitted from the upstream camera, and determines whether or not the absolute value is larger than the threshold value T2.

  Here, as described above, when the camera being processed is the v-th camera, the integration target code amount is obtained from the actual code amounts of all the cameras located upstream (that is, from 1 to v-1). It is calculated by subtracting a value obtained by multiplying the basic code amount by the number of cameras located upstream. The basic code amount is a value obtained by dividing a band that can be used for transmitting video within the communication band of the transmission path by the number of viewpoints (the number of cameras).

  As a supplement, when the camera being processed is the most upstream, the difference ΔUS in the integrated target code amount is set to zero. The threshold T2 may be determined according to the size of the buffer provided in the camera adapter 120. If it is determined that the absolute value of the accumulated target code amount difference ΔUS is larger than the threshold value T2, the camera adapter 120 shifts the process to step S1302, and if not, the process shifts to step S1303. Transition.

  In step 1302, the target code amount in the camera being processed is determined based on the following equation in consideration of ΔUS, which is the cumulative target code amount up to the upstream camera.

  By determining the target code amount based on the above equation, an attempt is made to reduce the code amount of the camera being processed by the band that has been used up to the upstream camera. Also, when it is determined that the absolute value of the difference ΔUS in the accumulated target code amount is equal to or less than the threshold T2 (No in S1301), in step S1303, the camera adapter 120, as in the first embodiment, Is set based on the following equation.

  In step S1304, the camera adapter 120 collates the visual field overlap map, and acquires the number of cameras capturing itself (ie, the visual field overlap degree) for each foreground. That is, the processing in step S1304 is the same processing as step S901 in FIG. 9 in the first embodiment.

  In step S1305, the camera adapter 120 acquires (measures) the number of foreground pixels captured in the connected camera 112 for each field overlap degree. That is, the processing in step S1305 is the same processing as step S903 in FIG. 9 in the first embodiment.

  In step S1306, the camera adapter 120 determines a quantization parameter to be applied to each foreground during encoding. That is, the processing in step S1306 is the same processing as step S904 in FIG. 9 in the first embodiment.

  In step S1307, the camera adapter 120 executes encoding processing using the quantization parameter for each foreground acquired up to the previous processing step. That is, the processing in step S1307 is the same processing as step S906 in FIG. 9 in the first embodiment.

  In step S1308, the camera adapter 120 updates the accumulated target code amount difference ΔUS based on the following equation.

  Based on the above formula, by updating the difference ΔUS in the accumulated target code amount, the difference between the target code amount and the actual code amount in the camera being processed is reflected in the difference ΔUS in the accumulated target code amount and passed downstream. Can do. The camera adapter 120 ends the code amount control process by executing the process in step S1308.

  With the configuration as described above, the difference between the target code amount and the actual code amount generated up to the immediately preceding camera can be absorbed by the camera being processed. As a result, the amount of code can be controlled so that the total daisy chain is within the bandwidth of the transmission path, so that transmission without delay is possible.

(Fourth embodiment)
In the first embodiment described above, when the actual code amount is excessive compared with the target code amount in the frame currently being processed, the encoding process is performed so as to compensate for the difference in the next frame. .

  However, if there is enough processing time to execute the encoding process, the actual code amount is set to the target code by increasing the quantization level and executing the encoding process again in the frame currently being processed. You may make it approach the quantity. By executing the encoding process in this way, the transmission waiting buffer of the camera adapter 120 can be saved.

(Fifth embodiment)
In the first embodiment described above, a field-of-view overlap map is created by executing a simulation before photographing as the preliminary data. Creating a field-of-view overlap map as prior data in this way is effective in a system in which a gazing point is determined before shooting and a camera is fixedly installed.

  However, the gaze point may be changed, for example, when shooting a plurality of competitions at the same stadium. In this case, the present invention can be implemented by updating the visual field overlap map at the time of executing calibration before photographing. In the case of a system or the like that can move a plurality of cameras in synchronization, the image processing system may be configured so that the visual field overlap map is updated each time dynamic calibration is executed.

(Sixth embodiment)
In the first embodiment described above, the field overlap / quantization parameter set correspondence table has been described as a parameter set for performing quantization according to a desired code amount. However, an option of “not transmitting” may be provided assuming that the band is more tight.

  In this case, as shown in FIG. 14, with respect to the field overlap / quantization parameter set correspondence table, as in the first embodiment, the quantization parameter increases stepwise from the one with the highest field overlap to the lower. Applied to be. In addition, as described above, assuming that the bandwidth is more tight, data transmission is not performed when the field overlap is 10 (dn = 10) and the quantization level is 7 (ql = 7). “Do not send” is set. That is, it is set so that the data is not encoded when the visual field overlap degree is equal to or greater than a predetermined value (that is, a predetermined number of cameras capturing the foreground).

  Note that, when “not transmit” is set for the quantization parameter setting, it means that the field overlap of the area is reduced by “1”. That is, when the quantization parameter is set to “not transmit” in the camera being processed, the signal is transmitted to the subsequent camera as a signal, and the visual field overlap degree is set to 1 for the region in the subsequent camera. Processing is performed after subtracting the number of units. As a supplement, in this case, with respect to the other foreground images, for example, unless the field overlap is 10 (dn = 10) and the quantization level is 7 (ql = 7), the field overlap is subtracted in the subsequent camera. No processing is performed. Further, by performing processing in this way, the minimum code amount of the foreground can be scaled up to 0, so that the range of code amount control can be expanded.

(Seventh embodiment)
In the first embodiment described above, an image (video) captured by all cameras on the transmission line is the target of code amount control. However, code amount control of video shot with a camera at an important position in the image quality of the virtual viewpoint content (for example, a camera shot at a predetermined angle with the gazing point as an axis, or a video shot with a telephoto camera, etc.) The image processing system can be configured not to be a target.

  In this case, the visual field overlap map created as the pre-data does not include the visual field of the camera at an important position in terms of the image quality of the virtual viewpoint content. Then, at the time of encoding, it is only necessary to skip the code amount control process of the camera and execute only the encoding process. With this configuration, it is possible to transmit a video captured by a camera with an important viewpoint in terms of the image quality of the virtual viewpoint content without deterioration, and generate a virtual viewpoint content.

(Other embodiments)
In the above-described embodiment, the example in which the camera adapters are connected in a daisy chain with one transmission path has been described. However, the present invention is not necessarily limited to this, and can be applied to other topologies. For example, a problem similar to that of a daisy chain also occurs in the bus type. In this case, the code amount may be controlled while acquiring information on camera adapters that are close to each other.

  In the case of the star type, the adjacent camera adapters are not directly connected, but the image processing system is configured so that necessary information such as the image quality of the adjacent camera adapters is obtained from the front-end server. do it.

  In the above-described embodiment, the example in which the image quality is set by setting the parameters for performing the encoding has been described, but the present invention is not necessarily limited thereto. For example, the image quality may be set by reducing the resolution or the image size or reducing the color gradation value. Moreover, although the example using a visual field duplication map was demonstrated as information which shows the number of the camera which image | photographs a predetermined object, the coordinate of the three-dimensional area | region used as imaging | photography object and the information of the camera which can image | photograph that coordinate are matched Other information such as information may be used.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

112 Camera 120 Camera adapter 180b Network 06130 Image processing unit

Claims (20)

In an image processing system that transmits images taken by a plurality of cameras through a predetermined transmission path and generates a virtual viewpoint image,
Camera number deriving means for deriving the number of cameras shooting a predetermined object;
An image quality setting means for setting an image quality of a foreground image including the predetermined object in accordance with the number of cameras capturing the predetermined object;
An image processing system comprising: Encoding means for encoding the foreground image based on the image quality of the foreground image set by the image quality setting means;
2. The image quality setting unit sets the image quality of the foreground image so that the number of cameras that capture the predetermined object and the image quality of the foreground image are inversely correlated. Image processing system.   The number-of-camera deriving means uses a field-of-view overlap map created by mapping the field-of-view overlap of the camera as a region from the three-dimensional shape of the field to be photographed and the relative positional relationship of each camera. The image processing system according to claim 1, wherein the number of cameras that shoot a predetermined object is derived.   The number-of-camera deriving means, in the field-of-view overlap map, when the predetermined object spans a plurality of areas where the camera field-of-view overlap is mapped, 4. The image processing system according to claim 3, wherein the number of cameras having the smallest number of cameras shooting the area is derived as the number of cameras shooting the predetermined object.   5. The apparatus according to claim 1, wherein the camera number deriving unit derives the number of cameras that are photographing the predetermined object using coordinates of a foot of the predetermined object. The image processing system described.   The image processing system according to claim 2, wherein the encoding unit quantizes the foreground image with a quantization parameter set based on an image quality of the foreground image.   The encoding means quantizes the foreground image with a quantization parameter set based on a target code amount set for each of the plurality of cameras and an image quality of the foreground image. Item 3. The image processing system according to Item 2.   8. The encoding unit according to claim 7, wherein the encoding unit calculates the target code amount by dividing a band for encoded data in a communication band of the transmission path by the number of the plurality of cameras. Image processing system. The encoding means includes
The basic code amount is calculated by dividing the band for encoded data in the communication band of the transmission path by the number of the plurality of cameras,
A value obtained by multiplying the integration target code amount by the actual code amount of all the cameras positioned upstream of one camera of the plurality of cameras by the number of cameras positioned upstream of the one camera. Is calculated by subtracting
9. The image processing according to claim 8, wherein when the integration target code amount is larger than a predetermined threshold, the target code amount is calculated by subtracting the integration target code amount from the basic code amount. system.   The encoding unit determines the quantization parameter of the next frame in the one camera according to a difference between the target code amount and an actual code amount of a current frame in one camera of the plurality of cameras. The image processing system according to claim 7, wherein the image processing system is an image processing system.   The image quality setting means sets the foreground image including the predetermined object not to be encoded when the number of cameras shooting the predetermined object is equal to or larger than the predetermined number. Item 11. The image processing system according to any one of Items 1 to 10.   The image quality setting means is set so as not to quantize a foreground image photographed by a camera arranged at a predetermined position in relation to the predetermined object among the plurality of cameras. The image processing system according to any one of 1 to 11.   The program for functioning a computer as an image processing system of any one of Claim 1 to 12. An image processing system control method for transmitting images taken by a plurality of cameras through a predetermined transmission path and generating a virtual viewpoint image,
A camera number deriving step for deriving the number of cameras shooting a predetermined object by the camera number deriving means;
An image quality setting step of setting the image quality of the foreground image including the predetermined object according to the number of cameras that are shooting the predetermined object by the image quality setting means;
A control method for an image processing system. In an image processing system for transmitting images taken by a plurality of cameras through a predetermined transmission path and generating a virtual viewpoint image, the transmission apparatus transmits the image to the predetermined transmission path.
Acquisition means for acquiring information indicating the number of cameras that photograph a predetermined object;
Image quality setting means for setting the image quality of the foreground image including the predetermined object based on the number of cameras indicated by the information acquired by the acquisition means;
Transmission means for transmitting the foreground image of the image quality set by the image quality setting means on the predetermined transmission path;
A transmission apparatus comprising: A changing means for changing the image quality of the foreground image based on the image quality set by the image quality setting means;
16. The transmission apparatus according to claim 15, wherein the transmission unit transmits the foreground image whose image quality has been changed by the changing unit through the predetermined transmission path. The changing means encodes the foreground image based on the image quality set by the image quality setting means,
17. The transmission apparatus according to claim 16, wherein the transmission unit transmits the foreground image encoded by the changing unit through the predetermined transmission path.   18. The information according to claim 15, wherein the information is acquired from a three-dimensional shape of imaging targets of the plurality of cameras and a degree of overlapping of the visual fields of the plurality of cameras based on a relative positional relationship between the cameras. 2. A transmission apparatus according to claim 1.   The transmission apparatus according to any one of claims 15 to 18, wherein the image quality setting means sets the image quality of the foreground image based on a transmission capacity of the predetermined transmission path. An image transmission method in an image processing system for transmitting images taken by a plurality of cameras through a predetermined transmission path and generating a virtual viewpoint image,
Based on the number of cameras that shoot a predetermined object, set the image quality of the foreground image including the predetermined object,
A transmission method comprising transmitting the foreground image having a set image quality through the predetermined transmission path.

Images