JP2018191251A - Information processing system, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress shortage of a communication band for transmission of a captured image.SOLUTION: When there is a restriction in a communication band and it is required to select data to be transmitted, camera adapters 120a to 120z determine to transmit the selected data, and store unselected data. Thereafter, the camera adapters 120a to 120z transmit the stored data when the communication band is available.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing method, and a program.

  There is a technique in which a plurality of cameras installed at different positions perform shooting synchronously, and virtual viewpoint content is generated using images taken from a plurality of viewpoints obtained by the shooting. According to such a technique, for example, since a highlight scene of soccer or basketball can be viewed from various angles, it is possible to give the user a higher sense of realism than a normal image.

  On the other hand, when generating and viewing virtual viewpoint content based on images taken from a plurality of viewpoints, first, images taken by a plurality of cameras are collected in an image processing unit such as a server. Next, the image processing unit generates a virtual viewpoint content by performing processing such as generation of a three-dimensional model and rendering using images taken by a plurality of cameras, and transmits the virtual viewpoint content to the user terminal.

  As a technique for aggregating images taken by a plurality of cameras, there is a technique described in Patent Document 1. Patent Document 1 discloses the following technique. That is, a plurality of cameras are connected to each other by an optical fiber via a control unit paired with the camera. The control unit accumulates image frames taken by a camera paired with the control unit. The image generation unit outputs an image expressing continuous motion using the image frames stored in each control unit.

US Pat. No. 7,106,361

To generate the virtual viewpoint content, images of objects photographed by a plurality of cameras are required. Therefore, when the number of objects to be imaged increases, the transmission load of the network can increase according to the number of objects. Therefore, when transmitting data such as images captured by each camera, there is a possibility that the communication band is insufficient depending on the system status, and data necessary for generating virtual viewpoint content cannot be transmitted.
The present invention has been made in view of such a problem, and an object thereof is to suppress a shortage of a communication band for transmitting a captured image.

  An information processing system according to the present invention is an adjustment unit that adjusts transmission target data that is data including an image of a subject included in a captured image and that is to be transmitted to an external device, and is adjusted by the adjustment unit. Storage means for storing data for making the transmission target data close to the original transmission target data in a storage medium, and transmission means for transmitting the data to the external device, wherein the transmission means includes the adjustment After transmitting the transmission target data adjusted by the means, data for transmitting the transmission target data adjusted by the adjusting means to be close to the original transmission target data is transmitted.

  ADVANTAGE OF THE INVENTION According to this invention, it can suppress that the communication band for transmission of a picked-up image is insufficient.

It is a figure which shows the structure of an image processing system. It is a figure which shows the structure of a camera adapter. It is a figure which shows the example of installation of an image processing system. It is a figure which shows the flow of the data between camera adapters. It is a flowchart explaining the process of a data selection process part. It is a figure which shows the 1st example of the time change of data amount. It is a figure explaining the 1st example of the change of the data transmitted to a database. It is a figure explaining the 2nd example of the change of the data transmitted to a database. It is a flowchart explaining the process of S5015 of FIG. It is a figure which shows the 2nd example of the time change of data amount.

Hereinafter, embodiments will be described in detail with reference to the drawings.
<First Embodiment>
First, the first embodiment will be described.
FIG. 1 is a diagram illustrating an example of the configuration of the image processing system 100. In this embodiment, a case where a plurality of cameras and a plurality of microphones are installed in a facility such as a stadium (stadium) or a concert hall will be described as an example. The image processing system 100 performs shooting and sound collection using a plurality of cameras and a plurality of microphones. The image processing system 100 includes sensor systems 110a to 110z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190. In the present embodiment, for example, an example of an information processing system is realized by the image processing system 100.

  The controller 300 includes a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management, parameter setting, and the like through the networks 310 a to 310 c, 180 a to 180 b, and 170 a to 170 y for each block constituting the image processing system 100. Here, the network may be a local area network, or may be configured by combining interconnects (such as Infiniband). Further, the network is not limited to these, and may be another type of network.

First, an example of an operation for transmitting images and sounds obtained from 26 sets of the sensor systems 110a to 110z from the sensor system 110z to the image computing server 200 will be described. In the present embodiment, a case where the sensor systems 110a to 110z are connected to each other in a daisy chain will be described as an example.
In the present embodiment, when there is no special description, the 26 sets of the sensor systems 110a to 110z are not distinguished and are described as the sensor system 110. Similarly, devices and networks in each sensor system 110 are described as a microphone 111, a camera 112, a camera platform 113, a camera adapter 120, and a network 170, unless they are specially described, unless otherwise described.

  In the present embodiment, the number of sensor systems 110 is 26 sets, but this is an example, and the number of sensor systems 110 is not limited to 26 sets. In the present embodiment, unless otherwise specified, an image is described as including a concept of a moving image and a still image. That is, the image processing system 100 according to the present embodiment can process both still images and moving images. In the present embodiment, an example in which the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual viewpoint sound will be mainly described. However, the virtual viewpoint content is not limited to this. For example, audio may not be included in the virtual viewpoint content. Further, for example, the sound included in the virtual viewpoint content may be a sound collected by a microphone located closest to the virtual viewpoint. Further, in the present embodiment, for the sake of simplification of explanation, detailed explanation of the sound is partially omitted, but the sound is basically processed together with the image.

Each of the sensor systems 110a to 110z includes one camera 112a to 112z. That is, the image processing system 100 includes a plurality of cameras for photographing the subject from a plurality of directions. The plurality of sensor systems 110 are connected to each other in a daisy chain.
The connection form of the plurality of sensor systems 110 is not limited to the daisy chain. For example, a star type network may be used. When a star network is used, all the sensor systems 110a to 110z are connected to the switching hub 180. In this case, data transmission / reception is performed between the sensor systems 110a to 110z via the switching hub 180.

  Further, FIG. 1 shows a configuration in which all of the sensor systems 110a to 110z are cascade-connected so as to be daisy chain connected. However, not all of the sensor systems 110a to 110z need be cascaded. For example, the plurality of sensor systems 110 may be divided into a plurality of groups. In this case, the sensor system 110 may be connected so as to be daisy chain connected in divided group units. Further, the camera adapter 120 that is the terminal of the divided group unit may be connected to the switching hub 180, and the switching hub 180 may output an image captured by each camera 112 to the image computing server 200. Such a configuration is particularly effective when the image processing system 100 is applied to a stadium. For example, a case where a stadium has a plurality of floors and the sensor system 110 is provided for each floor can be considered. If the above-described configuration is employed in such a case, for example, an image captured by the camera 112 can be input to the image computing server 200 for each floor or for each half of the stadium. Therefore, it is possible to simplify the installation of the sensor system 110 and make the system flexible even in a place where wiring for connecting all the sensor systems 110 with one daisy chain is difficult.

  Also, the control of image processing in the image computing server 200 is switched depending on whether the number of camera adapters 120 connected to each other in a daisy chain and outputting images to the image computing server 200 is one or more. It is done. That is, control of image processing in the image computing server 200 is switched depending on whether or not the sensor system 110 is divided into a plurality of groups. When there is one camera adapter 120 that outputs an image to the image computing server 200, an image is transmitted between the sensor systems 110 connected in a daisy chain, and an image is transmitted from the terminal camera adapter 120 to the image computing server 200. The Then, the image computing server 200 generates an image of the entire circumference of the stadium. For this reason, the timing at which the image data of the entire circumference of the stadium is gathered in the image computing server 200 is synchronized. That is, if the sensor system 110 is not divided into groups, the timing at which necessary image data is prepared in the image computing server 200 can be synchronized.

  On the other hand, when there are a plurality of camera adapters 120 that output images to the image computing server 200 (the sensor system 110 is divided into groups), the daisy chain lanes (routes) of the image data at the time of transmission The delay may be different. For this reason, the image computing server 200 needs to perform subsequent image processing while checking the concentration of the image data by waiting until the image data of the entire circumference of the stadium are gathered and synchronizing each image data. .

  In the present embodiment, the sensor system 110a includes a microphone 111a, a camera 112a, a pan head 113a, and a camera adapter 120a. Note that the configuration of the sensor system 110a is not limited to this configuration. The sensor system 110a only needs to include at least one camera adapter 120a and at least one camera 112a. For example, the sensor system 110a may be configured with one camera adapter 120a and a plurality of cameras 112a, or may be configured with one camera 112a and a plurality of camera adapters 120a. That is, at least one camera 112 and at least one camera adapter 120 in the image processing system 100 correspond to N to M (N and M are both integers of 1 or more). Further, the sensor system 110 may include devices other than the microphone 111a, the camera 112a, the pan head 113a, and the camera adapter 120a. Further, the camera 112a and the camera adapter 120a may be configured integrally. Furthermore, the front end server 230 may have at least a part of the functions of the camera adapter 120a. In the present embodiment, the configuration of the sensor systems 110b to 110z is the same as the configuration of the sensor system 110a, and thus a detailed description of the sensor systems 110b to 110z is omitted. The sensor systems 110b to 110z are not limited to the same configuration as the sensor system 110a, and may be configured differently.

  An image photographed by the camera 112a is subjected to image processing described later in the camera adapter 120a, and then transmitted to the camera adapter 120b of the sensor system 110b through the network 170a. The sound collected by the microphone 111a is also transmitted to the camera adapter 120b of the sensor system 110b via the network 170a. The sensor system 110b transmits the sound collected by the microphone 111b and the image taken by the camera 112b to the sensor system 110c together with the image and sound acquired from the sensor system 110a.

The operation described above is continued in the sensor systems 110c to 110z. As a result, the images and sounds acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z to the switching hub 180 via the network 180b, and then transmitted from the switching hub 180 to the image computing server 200.
In the present embodiment, a configuration in which the cameras 112a to 112z and the camera adapters 120a to 120z are separated is described as an example. However, these may be integrated in the same casing. In that case, the microphones 111a to 111z may be built in the cameras 112a to 112z integrated with the camera adapters 120a to 120z, or may be connected to the outside of the cameras 112a to 112z.

  Next, an example of the configuration and operation of the image computing server 200 will be described. The image computing server 200 of this embodiment processes data acquired from the sensor system 110z. The image computing server 200 includes a front end server 230, a database 250, a back end server 270, and a time server 290.

  The time server 290 has a function of distributing time and synchronization signals. The time server 290 distributes the time and the synchronization signal to the sensor systems 110a to 110z via the switching hub 180. The camera adapters 120a to 120z that have received the time and the synchronization signal Genlock the image data photographed by the cameras 112a to 112z based on the time and the synchronization signal to perform frame synchronization of the image data. That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112a to 112z. Further, the image processing system 100 generates a virtual viewpoint image based on a plurality of photographed images photographed at the same timing by adding information such as a time code to image data photographed by the plurality of cameras 112a to 112z. I can do it. For this reason, it is possible to suppress a decrease in the quality of the virtual viewpoint image due to a shift in shooting timing. In the present embodiment, it is assumed that the time server 290 manages time synchronization of the plurality of cameras 112. However, the time server 290 does not necessarily have to manage time synchronization of the plurality of cameras 112. For example, each camera 112 or each camera adapter 120 may perform processing for time synchronization independently.

  The front-end server 230 reconstructs a segmented transmission packet from the image data and audio data acquired from the sensor system 110z and converts the data format. The front-end server 230 writes the image data and audio data converted in data format in the database 250 in association with the camera identifier, data type, frame number, and the like.

  The database 250 manages each image data (frame) from each sensor system 110 acquired from the sensor system 110z and the reception status of the image data in a state management table. For example, the information management table stores a flag for each time and for each camera 112. The front-end server 230 sets a flag of 0 for each time and each camera 112 if the image data has not arrived and 1 if it has arrived. The information management table may store a flag for each predetermined time (for example, one second) and for each camera 112. In this case, the front-end server 230 sets a flag of 1 if all the image data has arrived from the camera 112 every predetermined time, and 0 if not.

  The back-end server 270 receives a virtual viewpoint designation from the virtual camera operation UI 330. The back-end server 270 reads image data and audio data corresponding to the viewpoint received from the virtual camera operation UI 330 from the database 250 based on the viewpoint received from the virtual camera operation UI 330. The back-end server 270 performs a rendering process on the image data and audio data read from the database 250 to generate a virtual viewpoint image. At this time, the database 250 provides image data and audio data to the back-end server 270 in response to a read request from the back-end server 270 in accordance with the reception status stored in the state management table.

  The rendered virtual viewpoint image and virtual viewpoint sound are transmitted from the back-end server 270 to the end user terminal 190. A user who operates the end user terminal 190 can view an image and view audio according to the designation of the viewpoint. That is, the back-end server 270 generates virtual viewpoint content based on images (a plurality of virtual viewpoint images) taken by a plurality of cameras 112 and virtual viewpoint information.

  The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the viewpoint designated by the virtual camera operation UI 330. The viewpoint may be specified by the user, or may be automatically specified based on the result of image analysis or the like.

Further, the back-end server 270 converts the virtual viewpoint image into the H.264 format. The data may be compressed and encoded by a standard technique represented by H.264 or HEVC and then transmitted to the end user terminal 190 using the MPEG-DASH protocol.
As described above, the image processing system 100 has three functional domains: a video collection domain, a data storage domain, and a video generation domain. The video collection domain includes sensor systems 110a-110z. The data storage domain includes a database 250, a front end server 230, and a back end server 270. The video generation domain includes a virtual camera operation UI 330 and an end user terminal 190. Note that the configuration of the image processing system 100 is not limited to such a configuration. For example, the virtual camera operation UI 330 can directly acquire image data from the sensor systems 110a to 110z. However, in the present embodiment, a data storage domain is arranged between the video collection domain and the video generation domain, instead of a method of directly acquiring image data from the sensor systems 110a to 110z. Specifically, the front-end server 230 converts image data, audio data, and meta information of these data generated by the sensor systems 110a to 110z into a common schema and data type of the database 250.

  In the present embodiment, the virtual camera operation UI 330 is accessed via the back-end server 270 without directly accessing the database 250. Common processing related to the image generation processing is performed by the back-end server 270, and processing of the difference portion of the application related to the operation UI is performed by the virtual camera operation UI 330. As a result, when developing the virtual camera operation UI 330, it is possible to focus on development of an operation device serving as a UI (user interface) and a UI function for operating a virtual viewpoint image to be generated. Further, the back-end server 270 can add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 330. This makes it possible to flexibly respond to requests from the virtual camera operation UI 330.

  As described above, in the image processing system 100, the virtual viewpoint image is generated by the back-end server 270 based on the image data captured by the plurality of cameras 112 for capturing the subject from a plurality of directions. Note that the image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be logically configured. The image processing system 100 may be configured by a plurality of devices or a single device.

  In this embodiment, for example, an example of a plurality of information processing apparatuses is realized by using the camera adapters 120a to 120z. Further, the virtual camera operation UI 330 sets a viewpoint for the subject. Further, the back-end server 270 uses the image of the subject area included in the plurality of photographed images photographed from the plurality of directions, and displays the image of the subject when viewed from the viewpoint set by the virtual camera operation UI 330. Generate.

(Camera adapter)
Next, an example of functional blocks of the camera adapter 120 in the present embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a functional configuration of the camera adapter 120. Details of the data flow between the functional blocks of the camera adapter 120 will be described later with reference to FIG.

The camera adapter 120 includes a network adapter 6110, a transmission unit 6120, an image processing unit 6130, and an external device control unit 6140.
The network adapter 6110 includes a data transmission / reception unit 6111 and a time control unit 6112.
The data transmission / reception unit 6111 performs data communication with other camera adapters 120, the front-end server 230, the time server 290, and the control station 310 via the networks 170, 291 and 310a. For example, the data transmission / reception unit 6111 outputs the foreground image and the background image separated by the foreground / background separation unit 6131 from the image taken by the camera 112 to another camera adapter 120. The camera adapter 120 that is the output destination of the foreground image and the background image is the camera adapter 120 in the next order of the camera adapter 120 in a predetermined order among the camera adapters 120 in the image processing system 100. Each camera adapter 120 outputs a foreground image and a background image, thereby generating a virtual viewpoint image based on the foreground image and the background image taken from a plurality of viewpoints. There may be a camera adapter 120 that outputs a foreground image separated from a captured image and does not output a background image. Further, the separation of the foreground image and the background image may be performed inside the image computing server 200 described later.

  The time control unit 6112 is based on, for example, the IEEE 1588 standard Ordinary Clock, and has a function of storing a time stamp of data transmitted to and received from the time server 290 and a function of performing time synchronization with the time server 290. Note that the protocol for time synchronization is not limited to IEEE 1588. For example, another EtherAVB standard or a unique protocol may be used. In the present embodiment, a network interface card (NIC) is used as the network adapter 6110. However, the network adapter 6110 is not limited to the NIC, and other interfaces may be used. IEEE 1588 has been updated as a standard, such as IEEE 1588-2002 and IEEE 1588-2008. The latter is also called PTPv2 (Precision Time Protocol Version 2).

The transmission unit 6120 has a function of controlling transmission of data to the switching hub 180 and the like via the network adapter 6110, and includes the following functional units.
The time synchronization control unit 6121 conforms to the IEEE 1588 standard PTP (Precision Time Protocol) and has a function of performing processing related to time synchronization with the time server 290. Note that the protocol for performing time synchronization is not limited to PTP, and time synchronization may be performed using other similar protocols.

  The image / audio transmission processing unit 6122 has a function of creating a message for transferring image data and audio data to another camera adapter 120 or the front-end server 230 via the data transmission / reception unit 6111. The message includes image data and audio data, and meta information of each data. The meta information of this embodiment includes a time code or sequence number when an image is taken and when audio is sampled, a data type, an identifier indicating the individual of the camera 112 and the microphone 111, and the like. The image / sound transmission processing unit 6122 receives a message from another camera adapter 120 via the data transmission / reception unit 6111. Then, the image / audio transmission processing unit 6122 converts the data information fragmented into the packet size specified in the transmission protocol into image data and audio data according to the data type included in the message received from the other camera adapter 120. Restore.

The bandwidth monitoring unit 6123 monitors the amount of data transmitted from the upstream side of the network 170 rather than the camera adapter 120 to which it belongs, and notifies the image processing unit 6130 described later of the result.
The image processing unit 6130 has a function of processing image data captured by the camera 112 under the control of the camera control unit 6141, and includes the following functional units.

  The foreground / background separator 6131 has a function of separating image data captured by the camera 112 into a foreground image and a background image. That is, the foreground / background separation unit 6131 extracts a predetermined area from the image data photographed by the camera 112 corresponding to the camera adapter 120 to which the foreground / background separation unit 6131 belongs. The predetermined area is, for example, an area of the foreground image obtained as a result of object detection for the captured image. The foreground / background separator 6131 separates the captured image into a foreground image and a background image based on the extraction result of the predetermined area. Note that an object is, for example, a subject such as a person. However, the object may be a specific person (player, manager, and / or referee, etc.), or may be an object with a predetermined image pattern, such as a ball or a goal. A moving object may be detected as the object. An image of a portion corresponding to an object of a virtual viewpoint image generated in the image processing system 100 by separating and processing a foreground image including an important object such as a person and a background area not including such an object Quality can be improved. Further, the foreground image and the background image are separated by each of the plurality of camera adapters 120, whereby the load on the image processing system 100 including the plurality of cameras 112 can be distributed. The predetermined area is not limited to the foreground image, and may be a background image, for example.

  The data selection processing unit 6132 measures the data amount of the foreground image and the background image separated by the foreground / background separation unit 6131. The data selection processing unit 6132 determines whether or not the sum of the measured data amount and the data amount of data transmitted from the network 170 upstream of the camera adapter 120 to which the data selection unit 6 belongs exceeds a predetermined data amount. . Note that the bandwidth monitoring unit 6123 notifies the data amount of data transmitted from a network upstream from the camera adapter 120 to which the camera adapter 120 belongs.

  As a result of this determination, when the data amount exceeds the predetermined data amount, the data selection processing unit 6132 limits the data amount to the predetermined data amount or less. For this purpose, first, the data selection processing unit 6132 selects data to be transmitted from the foreground image and background image separated by the foreground / background separation unit 6131. Then, the data selection processing unit 6132 sends the selected data to the transmission unit 6120 and sends data that has not been sent to the storage unit 6133. On the other hand, when the data amount does not exceed the predetermined data amount, the data selection processing unit 6132 reads out the data stored in the storage unit 6133, adds it, and sends it to the transmission unit 6120. Details of the processing of the data selection processing unit 6132 will be described later.

The external device control unit 6140 has a function of controlling devices connected to the camera adapter 120, and includes the following functional units.
The camera control unit 6141 is connected to the camera 112 and has functions of controlling the camera 112, obtaining image data captured by the camera 112, providing a synchronization signal, and setting a time.

The control of the camera 112 includes, for example, setting and reference of shooting parameters (such as setting of the number of pixels, color depth, frame rate, and white balance). Furthermore, the control of the camera 112 includes acquisition of the state of the camera 112 (during shooting, stop, synchronization, error, etc.), start and stop of shooting, and focus adjustment.
The synchronization signal is provided by providing the camera 112 with a photographing timing (control clock) using a time synchronized with the time server 290 obtained by the time synchronization control unit 6121.
The time setting is performed by providing the time synchronized with the time server 290 obtained by the time synchronization control unit 6121 using a time code conforming to the format of SMPTE12M, for example. As a result, the time code provided from the camera control unit 6141 is added to the image data received from the camera 112. Note that the format of the time code is not limited to SMPTE12M, and other formats may be used. Further, the camera control unit 6141 may provide the time code to the image data received from the camera 112 without providing the time code to the camera 112.

The microphone control unit 6142 is connected to the microphone 111 and has a function of controlling the microphone 111, starting and stopping sound collection by the microphone 111, obtaining audio data collected by the microphone 111, and the like.
The pan head control unit 6143 is connected to the pan head 113 and has a function of controlling the pan head 113. The control of the pan head 113 includes, for example, pan / tilt control, acquisition of the state of the pan head 113, and the like.
In the image processing system 100 of the present embodiment, the foreground image and the background image are transmitted between the plurality of camera adapters 120 connected in a daisy chain and input to the front end server 230. Here, when the foreground area is extremely large in the captured image, the amount of data of the foreground image to be transmitted becomes enormous.

  An example in which the image processing system 100 of this embodiment is installed in a soccer stadium will be described with reference to FIG. In FIG. 3, the network 170 that connects the camera adapters 120 in a daisy chain is divided into two systems, networks 170A and 170B. The cameras 112 corresponding to the camera adapters 120 connected to the respective networks 170A and 170B are installed so as to capture the attention area 3010A or 3010B so as to cover the front of each goal. In the example shown in FIG. 3, one image processing system 100 is installed for each of the attention areas 3010A and 3010B. As a characteristic of group sports such as soccer, the number of players located on the side where the ball 3020 is located increases. For this reason, in the sensor system 110 that captures the attention area 3010A on the side where the ball 3020 is located, the number of objects that become foreground images increases, and the amount of data to be transmitted increases. On the other hand, the number of players located on the side without the ball 3020 is reduced. For this reason, in the sensor system 110 that captures the attention area 3010B on the side without the ball, the number of objects that become foreground images decreases, and the amount of data to be transmitted decreases. When the number of players in the attention area increases, the amount of data output from the sensor system 110 that captures the attention area increases. For this reason, when the sensor system 110 transmits data as it is, the sensor system 110 connected downstream of the network 170 connected in a daisy chain saturates the communication band with the data transmitted from the upstream, and transmits the data. It becomes impossible.

Therefore, in this embodiment, when players gather in the attention area and the amount of data output from the sensor system 110 increases, the amount of data transmitted in the daisy chain exceeds the preset transmission band. Do not. For example, by assigning a transmittable band for each sensor system 110, the amount of data is prevented from exceeding a preset transmission band. An example of this method will be described with reference to FIGS.
FIG. 4 is a diagram illustrating an example of a data flow between the camera adapters 120a, 120b, and 120c. The camera adapter 120a and the camera adapter 120b are connected to each other, and the camera adapter 120b and the camera adapter 120c are connected to each other. A camera 112b is connected to the camera adapter 120b, and the camera adapter 120c is connected to the front-end server 230. Hereinafter, an example of data output processing of the image processing unit 6130 of the camera adapter 120b will be described.

  The transmission unit 6120 of the camera adapter 120b includes data obtained by performing image processing and data amount adjustment on the image data 6720 captured by the camera 112b and data 6721 transmitted from the camera adapter 120a. Are entered. In addition, meta information including a time code for the image data 6720 is input to the transmission unit 6120. The transmission unit 6120 performs processing such as packetization on the input image data and meta information, and outputs the processed data to the network adapter 6110.

  Image data 6720 photographed by the camera 112b illustrated in FIG. 4 is an example of data including an image of a subject included in a photographed image photographed by a photographing apparatus connected to the information processing apparatus. In addition, for example, the camera adapter 120a realizes an example of the information processing apparatus that is in the order immediately preceding the information processing apparatus in a predetermined order. In addition, for example, an example of data including an image of a subject transmitted from the information processing apparatus that is in the order immediately preceding the information processing apparatus in a predetermined order based on the data 6721 transmitted from the camera adapter 120a. Realized. In addition, for example, the camera adapter 120c realizes an example of the information processing apparatus that becomes the next order of the information processing apparatus in a predetermined order.

Next, an example of processing of the data selection processing unit 6132 when selecting data to be output from the camera adapter 120b will be described with reference to the flowchart of FIG. The flowchart of FIG. 5 is repeatedly executed for each data output from the camera adapter 120b.
The data selection processing unit 6132 acquires the amount of data output from the foreground / background separation unit 6131 (S5001). Next, the data selection processing unit 6132 acquires the data amount of the data 6721 transmitted from the camera adapter 120a from the bandwidth monitoring unit 6123 (S5002). Next, the data selection processing unit 6132 derives the data amount of data to be output to the camera adapter 120c based on the data amount acquired in S5001 and S5002 (S5003).

  In the present embodiment, the data selection processing unit 6132 determines the data amount of the transmission target data that is data including the image of the subject included in the photographed image and to be transmitted to the external device in S5001 and S5002. get. The transmission target data is, for example, data output from the foreground / background separation unit 6131 and data 6721 transmitted from the camera adapter 120a.

Next, the data selection processing unit 6132 compares the data amount derived in S5003 with the transmission band restriction amount specified in advance, and confirms the transmission possibility. Specifically, the data selection processing unit 6132 determines whether or not the amount of data to be output to the transmission unit 6120 exceeds a transmission band restriction amount specified in advance (S5004). Note that the comparison between the data amount derived in S5003 and the transmission bandwidth restriction amount specified in advance may be performed for each data type (here, foreground image, background image, for example).
In the present embodiment, the data selection processing unit 6132 determines in S500 whether the data amount of the transmission target data exceeds a threshold value. The case of Yes in S5004 is an example when the data amount of the transmission target data exceeds a threshold value, and the case of No in S5004 is an example of the case where the data amount of the transmission target data does not exceed the threshold value. . Further, for example, an example of a threshold value is realized by a transmission band restriction amount.

  As a result of the determination in S5004, when the amount of data derived in S5003 exceeds the transmission bandwidth constraint amount specified in advance (in the case of Yes in S5004), the data selection processing unit 6132 sets the processing method when the output data amount exceeds The selection setting is acquired (S5005). This selection setting may be set from the controller 300. In addition, the selection setting may be determined according to the type of the target end user terminal 190 or the type of competition. For example, if the type of the end user terminal 190 is a type with a low resolution, a process for reducing the resolution is determined as the selection setting. The data selection processing unit 6132 selects one of a plurality of processes described below based on the selection setting of the processing method when the output data amount exceeds that acquired in S5005 (S5006).

  When the selection setting indicates a setting for reducing the frame rate, the data selection processing unit 6132 performs processing for reducing the frame rate for outputting the image data and outputting it to the transmission unit 6120 (S5007). In this process, the data amount is reduced by thinning and transmitting frames including the foreground image and the background image. The transmission unit 6120 transmits the data processed in S5007 via the network adapter 6110. Then, the data selection processing unit 6132 stores the thinned frame data together with the meta information in the storage unit 6133 (S5008).

When the selection setting indicates a setting for reducing the resolution, the data selection processing unit 6132 performs a process of reducing the resolution of the image data and outputting it to the transmission unit 6120 (S5009). The transmission unit 6120 transmits the data processed in S5009 via the network adapter 6110. Then, the data selection processing unit 6132 stores the difference between the image data before the resolution is lowered and the image data after the resolution is lowered in the storage unit 6133 together with the meta information (S5010).
When the selection setting indicates a setting for performing compression processing, the data selection processing unit 6132 performs processing for compressing the image data and outputting it to the transmission unit 6120 (S5011). The transmission unit 6120 transmits the data processed in S5011 via the network adapter 6110. The data selection processing unit 6132 normally performs lossless compression to reduce the data amount of the image data. However, when the data amount derived in S5003 exceeds the transmission bandwidth constraint amount specified in advance, in order to increase the compression rate. Switch to lossy compression to reduce the amount of data. When the lossy compression is performed, the data selection processing unit 6132 stores information (for example, an uncompressed image) for restoring the original image in the storage unit 6133 together with the meta information (S5012). The information for restoring the original image is not limited to this, and may be information obtained by encoding data for correcting the deterioration as a result of irreversible compression. The data selection processing unit 6132 may store the reversibly compressed image in the storage unit 6133.

  When the selection setting indicates a setting for selecting image data with high importance, the data selection processing unit 6132 performs the following processing. That is, the data selection processing unit 6132 performs processing for selecting at least one of the foreground image and the background image to be transmitted according to the importance of the foreground image and the background image and outputting the selected image together with the meta information to the transmission unit 6120 ( S5013). The transmission unit 6120 transmits the data processed in S5013 via the network adapter 6110. Then, the data selection processing unit 6132 stores the image data of the foreground image / background image not selected in S5013 in the storage unit 6133 together with the meta information (S5014).

  In the present embodiment, the data selection processing unit 6132 adjusts the transmission target data in S5007, S5009, S5011, and S5013. In S5007, S5009, and S5013, data to be transmitted to the external device is selected from the transmission target data. Also, for example, examples where S5007, S5009, and S5013 select data to be transmitted to the external device from the transmission target data in units of frames, pixels, and regions determined based on objects, respectively. It is. The regions determined based on the objects correspond to, for example, foreground image and background image regions. Further, for example, S5011 is an example of compressing the transmission target data. In this embodiment, the data selection processing unit 613 stores data for bringing the adjusted transmission target data closer to the original transmission target data in S5008, S5010, S5012, and S5014.

  The processing in the data selection processing unit 6132 and the data stored in the storage unit 6133 are not limited to the example described above. For example, the data selection processing unit 6132 performs a selection process that suppresses the amount of data to be transmitted within a threshold value N of a data amount that is specified in advance, and the transmitted data can be restored to the data before selection or can be restored as much as possible. The data may be stored in the storage unit 6133 together with the meta information. In addition, the data selection processing unit 6132 may store all of the original data in the storage unit 6133 when restoration using the transmitted data is impossible.

  On the other hand, when the data amount derived in S5003 does not exceed the transmission bandwidth restriction amount specified in advance (No in S5004), the data selection processing unit 6132 performs the following processing. In other words, the data selection processing unit 6132 reads out from the storage unit 6133 data that falls within the difference between the data amount derived in S5003 and the transmission bandwidth constraint amount specified in advance (S5015). Then, the data selection processing unit 6132 outputs the read data to the transmission unit 6120. The transmission unit 6120 transmits, via the network adapter 6110, the data read in S5015 and the data that does not exceed the transmission bandwidth restriction amount (data whose data amount has been acquired in S5001 and S5002).

  In the present embodiment, for example, the transmission unit 6120 and the network adapter 6110 transmit the adjusted transmission target data in S5015, and then make the adjusted transmission target data close to the original transmission target data. Is transmitted.

  FIG. 6 is a diagram illustrating an example of a temporal change in the data amount. Specifically, FIG. 6 is a diagram illustrating an example of a time change of a value obtained by adding the data amount of data output from the foreground / background separation unit 6131 and the data amount of data transmitted from the upstream side of the network. In FIG. 6, the transmission bandwidth restriction amount designated in advance is represented by N. The transmission band restriction amount N in each camera adapter 120 can be set to the following value, for example. That is, a value obtained by dividing the transmission amount in the daisy chain by the number of sensor systems 110 installed in the image processing system 100 is multiplied by the number of sensor systems 110 connected upstream of the daisy chain +1. I can do it. However, the transmission band restriction amount N in each camera adapter 120 is not limited to this. The transmission amount in the daisy chain is a predetermined amount of data that can be transmitted to the network, for example, the maximum amount of data that can be transmitted to the network.

  If the data amount of data output from each camera adapter 120 exceeds the transmission bandwidth restriction amount N specified in advance, if the data is output as it is, the communication bandwidth may be saturated on the downstream side of the daisy chain. high. For this reason, the sensor system 110 connected downstream of the network cannot transmit data. Therefore, the data selection processing unit 6132 selects data to be output so that the data amount of data output from the camera adapter 120 to which the data selection processing unit 6132 belongs is within the transmission bandwidth restriction amount N. In this way, the data output from the data selection processing unit 6132 to the transmission unit 6120 is reduced from the data 610 to the data 620. Then, the data selection processing unit 6132 stores the data not output from the camera adapter 120 to which the data selection processing unit 6132 belongs in the storage unit 6133. If the data selection processing unit 6132 determines that the amount of data to be output does not exceed the transmission bandwidth restriction amount N designated in advance, the data selection processing unit 6132 uses this vacant band (shaded portion in FIG. 6) to store the data. Data is read from 6133 and the data is transmitted together.

Next, changes in data transmitted to the database 250 as a result of the above-described data selection process will be described with reference to FIGS. 7 and 8, including the passage of time.
First, referring to FIG. 7, the data derived by the data selection processing unit 6132 exceeds the transmission bandwidth restriction amount N specified in advance, and therefore the case where the data is transmitted with the frame rate reduced to a quarter is taken as an example. As an example, an example of changes in data transmitted to the database 250 will be described. This description is a description of the processing when the processing of S5007 is selected in S5006 of FIG. In FIG. 7, A, B, C, and D represent the sensor system 110, and t1, t2, t3, t4, and t5 represent time. Circles represent frame data captured by the camera 112 at times t1 to t5.

  The data selection processing unit 6132 determines that the amount of data to be output to the transmission unit 6120 exceeds the transmission bandwidth restriction amount N when the image is captured by the camera 112 connected to the camera adapter 120 to which the data selection unit 6132 belongs (Yes in S5004). See). The data selection processing unit 6132 selects frame data represented by a solid circle in the state 710 in FIG. 7 (S5007), and outputs the frame data to the transmission unit 6120. Further, the data selection processing unit 6132 stores the frame data represented by the broken-line circle in the state 710 in FIG. 7 in the storage unit 6133 (S5008). As a result, the data amount of data output from the data selection processing unit 6132 to the transmission unit 6120 is reduced to a quarter of the data amount of data originally output from the data selection processing unit 6132 to the transmission unit 6120.

  After that, when the data selection processing unit 6132 determines that the data amount of the data to be output to the transmission unit 6120 does not exceed the transmission bandwidth restriction amount N (No in S5004), the frame data represented by the dotted circle stored in the storage unit 6133 Is read (S5015). Then, the data selection processing unit 6132 adds the read frame data to the transmission unit 6120 in addition to the data output from the foreground / background separation unit 6131. As a result, the frame data (frame data represented by a solid circle) transmitted from the transmission unit 6120 gradually increases with time (see state 710 → state 720 → state 730 → state 740 in FIG. 7). . Finally, all the frame data is transmitted from the transmission unit 6120. Actually, there is a variation between the sensor systems 110 in the time until transmission of frame data transmitted when there is a vacant transmission band. In the example shown in FIG. 7, the data selection processing unit 6132 selects frame data to be output to the transmission unit 6120 in the order of frame data at times t1 and t5 → frame data at t3 → frame data at t2 → frame data at t4. To do. In this way, the frames for which transmission has been completed can be made as even as possible during transmission of frame data.

  In the database 250, whether or not the reception of one frame of frame data from each sensor system 110 at each time is completed and whether or not the frame data of all the sensor systems 110 at that time are available are stored in the state management table. to manage. In response to the frame data request from the back-end server 270, the database 250 determines whether or not to provide the frame data along the state management table. For this reason, the virtual viewpoint image generated by the back-end server 270 is an image of 15 fps immediately after shooting, but as time passes, the frame rate gradually increases as the data is gathered, and finally the image of 60 fps. It becomes a virtual viewpoint image.

  Next, referring to FIG. 8, an example of a case where the data derived by the data selection processing unit 6132 exceeds the transmission bandwidth restriction amount N specified in advance, and the data is transmitted with the resolution reduced to a half length and width. An example of changes in data transmitted to the database 250 will be described. Note that this description is an explanation of the processing when the processing of S5009 is selected in S5006 of FIG. In FIG. 8, x1 to x5 represent the x coordinate of the pixel, y1 to y5 represent the y coordinate of the pixel, and the circle represents the pixel data of each coordinate.

  The data selection processing unit 6132 determines that the amount of data to be output to the transmission unit 6120 exceeds the transmission bandwidth restriction amount N when the image is taken by the camera 112 connected to the camera adapter 120 to which it belongs (Yes in S5004). reference). Then, the data selection processing unit 6132 selects pixel data represented by a solid circle in the state 810 in FIG. 8 (S5009), and outputs it to the transmission unit 6120. Further, the data selection processing unit 6132 stores the pixel data represented by the broken-line circle in the state 810 of FIG. 8 in the storage unit 6133 (S05010). As a result, the data amount of data output from the data selection processing unit 6132 to the transmission unit 6120 is reduced to a quarter of the data amount of data originally output from the data selection processing unit 6132 to the transmission unit 6120.

  Thereafter, when the data selection processing unit 6132 determines that the data amount of the data to be output to the transmission unit 6120 does not exceed the transmission band restriction amount N (No in S5004), the pixel data represented by the dotted circle stored in the storage unit 6133 Read (S5015). Then, the data selection processing unit 6132 adds the read pixel data to the transmission unit 6120 in addition to the data output from the foreground / background separation unit 6131. As a result, pixel data (pixel data represented by a solid circle) transmitted from the transmission unit 6120 gradually increases as time passes (see state 810 → state 820 → state 830 → state 840 in FIG. 8). . Finally, all the pixel data is transmitted from the transmission unit 6120. Actually, the time required to complete transmission of pixel data transmitted when there is a vacant transmission band varies among sensor systems 110. In the example illustrated in FIG. 8, the data selection processing unit 6132 includes pixel data of odd columns / odd rows → pixel data of even columns / even rows → pixel data of even columns / odd rows → pixel data of odd columns / even rows. In order, pixel data to be output to the transmission unit 6120 is selected. In this way, it is possible to make the pixels for which transmission has been completed as uniform as possible during the transmission of the pixel data.

  In the database 250, the reception status of pixel data of each frame from each sensor system 110 is managed by a state management table. When there is insufficient pixel data in response to the frame data request from the back-end server 270, the database 250 complements the necessary pixels along the state management table, and stores the pixel data. provide. For this reason, the virtual viewpoint image generated by the back-end server 270 has a vertical and horizontal resolution that has been reduced to one-half immediately after shooting, but the resolution gradually increases as time passes to gather the data. Eventually, a virtual viewpoint image with the original resolution is obtained.

7 and 8, the case where the amount of data is reduced to a quarter has been described as an example for easy understanding. However, the amount of data to be reduced is determined by how much the data amount of data to be output to the transmission unit 6120 exceeds the transmission band restriction amount N (data amount that can be transmitted by the camera adapter 120).
In the present embodiment, the network 170 connected in a daisy chain has been described as an example. However, the connection form of the camera adapter 120 is not limited to the daisy chain connection. For example, even in a star type network, the present invention can be applied to a case where it is necessary to limit the amount of data output from each sensor system 110. For example, there may be a case where there is an upper limit in the bandwidth of data input from the network of the front-end server 230, or a case where there is an upper limit in the bandwidth of data input in the database 250. In this case, the method of the present embodiment can be applied in the same manner as in the case of the daisy chain connection described above, except that it is not necessary to monitor the data amount of the upstream data with the bandwidth monitoring unit 6123. is there.

  In the present embodiment, the storage unit 6133 has been described as having a sufficient capacity. However, the storage unit 6133 may be in a situation where a certain amount of capacity must be used. In this case, the data selection processing unit 6132 needs to delete the contents of the storage unit 6133 and store new image data. For this reason, the data selection processing unit 6132 sequentially deletes the oldest time code, for example. However, the method for storing necessary data in the storage unit 6133 when the capacity is limited is not limited to this. For example, the data selection processing unit 6132 may determine data to be deleted from the storage unit 6133 based on the importance of the data in the screen, such as deleting from the background image.

  As described above, in the present embodiment, the data selection processing unit 6132 determines that the selected data is to be transmitted and stores the unselected data when there is a restriction on the communication band and it is necessary to select the data to be transmitted. Store in the unit 6133. Thereafter, the data selection processing unit 6132 transmits the data stored in the storage unit 6133 when there is a free communication band. Therefore, when generating a virtual viewpoint image, it is possible to suppress a shortage or compression of a communication band for transmitting an image captured by each camera 112. In addition, it is possible to gradually generate a high-quality virtual viewpoint image as time passes.

  In the present embodiment, the case where the processing method is selected when the data amount of the data output from the camera adapter 120 exceeds the transmission bandwidth restriction amount N designated in advance has been described. The method is not limited. It is not necessary to select all the processes of S5007 to S5008, S5009 to S5010, S5011 to S5012, and S5013 to S5014 shown in FIG. 5, and any one of them may be performed. Moreover, a process is not limited to these.

<Second Embodiment>
Next, a second embodiment will be described. In the first embodiment, the case where the communication band is monitored for each sensor system 110 and the data to be transmitted is determined when there is a vacant communication band has been described as an example. However, when the communication capacity is small with respect to the data capacity stored in the storage unit 6133, there is a case where the priority order of the data to be transmitted is instructed and the data with higher priority is desired to be transmitted. In this embodiment, such a case will be described. As described above, in the present embodiment and the first embodiment, the data selection processing unit 6132 determines that the data amount of the data to be output to the transmission unit 6120 does not exceed the transmission band restriction amount N (No in S5004). The processing is different mainly. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS.

An example of the configuration of the image processing system of this embodiment is the same as the configuration shown in FIG. The controller 300 selects and determines a scene that is determined to be important by the operator, or automatically analyzes an image to determine a scene that is determined to be important. For example, when an image processing system is applied to a soccer stadium, an important scene here refers to a scene that the user wants to look back, such as a goal scene or a shot scene. In the present embodiment, a case where a scene is designated by time code (start time and end time) at the time of shooting will be described as an example.
The scene information determined by the controller 300 in this way is notified to the data selection processing unit 6132 in the sensor system 110 via the networks 310a, 180a, and 170.
In the present embodiment, the data selection processing unit 6132 acquires scene information.

  Next, an example of the process of the data selection processing unit 6132 (the process of S5015 in FIG. 5) when the data communication band is free will be described with reference to FIGS. FIG. 9 is a flowchart for explaining an example of the process of S5015 of FIG. FIG. 10 is a diagram illustrating an example of a temporal change in the data amount. Specifically, FIG. 10 is a diagram illustrating an example of a time change of a value obtained by adding the data amount of data output from the foreground / background separation unit 6131 and the data amount of data transmitted from the upstream side of the network. In FIG. 10, the transmission band restriction amount designated in advance is represented by N.

  Here, when the data amount exceeds the transmission bandwidth restriction amount N, the data selection processing unit 6132 performs data selection processing, and the data not selected in this selection processing is stored in the storage unit 6133. In FIG. 10, the periods stored in the storage unit 6133 are represented by D1 to D4, and the periods in which there is a margin in the data communication band are represented by E1 to E3. In addition, the controller 300 designates the following two pieces of scene information as scene information for specifying an important scene. First, the controller 300 designates scene information P1. The scene information P1 includes the start time t1s and the end time t1e of the first important scene. Further, the controller 300 designates the scene information P2. The scene information P2 includes the start time t2s and the end time t2e of the next important scene.

  In periods D1 (time t0 to time t1), D2 (time t2 to time t3), D3 (time t4 to time t5), and D4 (time t6 to time t7), the data amount exceeds the transmission band constraint amount N. Therefore, in S5008, S5010, S5012, and S5014, the data selection processing unit 6132 adds the time information and stores the data in the storage unit 6133. Next, in a period E1 (time t1 to time t2) in which the data amount is less than the transmission band restriction amount N, no scene information is instructed from the controller 300. For this reason, the data selection processing unit 6132 determines that the scene information is not instructed from the controller 300 (determined No in S9001 of FIG. 9). Therefore, the data selection processing unit 6132 reads out the data stored in the period D1 from the storage unit 6133 and transmits the data (see S9003 in FIG. 9). Since the amount of data that can be transmitted in the period E1 is smaller than the amount of data stored in the period D1, the data stored in the period D1 still remains in the storage unit 6133 at the time t2. Next, in the period D2 (time t2 to time t3), the data amount again exceeds the transmission band restriction amount N. Therefore, the data selection processing unit 6132 adds the time information and stores the data in the storage unit 6133. Here, scene information (start time: t1s, end time: t1e, scene information P1) to be transmitted with priority from the controller 300 is notified.

  When the data amount falls below the transmission band restriction amount N at time t3 (when it is determined No in S5004), the data selection processing unit 6132 determines that the scene information is instructed (Yes in S9001 in FIG. 9). Determined). Therefore, the data selection processing unit 6132 preferentially stores the data indicated by the scene information in the order stored in the storage unit 6133 (that is, the oldest in time order), not the data stored in the period D1. The data is read from the unit 6133 (steps S9002 and S9003 in FIG. 9). Specifically, data from time t1s to time t1e is read from the storage unit 6133. In the present embodiment, the data selection processing unit 6132 selects data specified by the data specifying information from the stored data in S9002.

  After the reading of the data instructed by the scene information is completed, the data selection processing unit 6132 reads the data stored in the storage unit 6133 again in the order in which the storage to the storage unit 6133 was performed. Also at time t7, similarly to time t3, the data selection processing unit 6132 determines that the data amount is less than the transmission bandwidth restriction amount N (determined No in S5004). In this case, the data selection processing unit 6132 preferentially reads the data instructed by the scene information (data from time t2s to time t2e) from the storage unit 6133 (steps S9002 and S9003 in FIG. 9).

  As described above, in the present embodiment, the data selection processing unit 6132 preferentially selects scene data designated from the controller 300 and outputs the data to the transmission unit 6120 when there is an empty data communication band. Accordingly, in addition to the effects described in the first embodiment, it is possible to generate a virtual viewpoint image of a scene with higher priority with a smaller delay of time.

  In the present embodiment, the case where the scene is designated by the time code (start time and end time) at the time of shooting has been described as an example. However, the method for designating the scene is not limited to this, and it is sufficient that the scene can be specified. For example, the controller 300 may specify a frame number. Further, the controller 300 gives priority (priority) information for each scene specified by the scene information for specifying an important scene so that a scene with higher importance can be transmitted with priority. Can be added. In addition, also in the present embodiment, various modifications described in the first embodiment can be employed. In the present embodiment, for example, an example of time information at which data is obtained is realized by using a time code at the time of shooting, and an example of identification information of data is realized by using a frame number. In addition, for example, by using the scene information P1 and P2, an example of information for specifying a plurality of data is realized. Further, for example, by using information on the priority (priority order) added to each scene specified by the scene information, an example of information indicating the priority order for the stored data is realized.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  110a to 110z: sensor system, 112a to 112z: camera, 120a to 120z: camera adapter, 170a to 170y: network

Claims (24)

Adjustment means for adjusting transmission target data, which is data including an image of a subject included in a captured image, and is data to be transmitted to an external device;
Storage means for storing, in a storage medium, data for bringing the transmission target data adjusted by the adjustment means closer to the original transmission target data;
Transmission means for transmitting data to the external device,
The transmission means transmits the data for adjusting the transmission target data adjusted by the adjustment means to be close to the original transmission target data after transmitting the transmission target data adjusted by the adjustment means. Information processing system. It further has a processing means for processing the captured image,
The information processing system according to claim 1, wherein the transmission target data includes data processed by the processing unit and data received from an external device.   The information processing system according to claim 1, wherein the adjustment unit adjusts the transmission target data when a data amount of the transmission target data exceeds a threshold value.   The said transmission means transmits the said transmission object data adjusted by the said adjustment means, when the data amount of the said transmission object data exceeds a threshold value, The any one of Claims 1-3 characterized by the above-mentioned. Information processing system.   The transmission unit transmits data including the transmission target data and at least a part of the data stored in the storage unit to the external device when a data amount of the transmission target data does not exceed a threshold value. The information processing system according to claim 1, wherein the information processing system is an information processing system.   The transmission means transmits data for bringing the transmission target data adjusted by the adjustment means closer to the original transmission target data at a later timing than the transmission target data adjusted by the adjustment means. The information processing system according to any one of claims 1 to 5, wherein when the data amount of the transmission target data does not exceed a threshold value, the data is transmitted together with the transmission target data whose data amount does not exceed the threshold value.   The data for bringing the transmission target data adjusted by the adjustment unit closer to the original transmission target data is data for restoring the transmission target data adjusted by the adjustment unit to the original transmission target data. The information processing system according to claim 1, wherein the information processing system is provided.   The information processing system according to claim 1, wherein the adjustment unit selects data to be transmitted to the external device from the transmission target data.   9. The information processing system according to claim 8, wherein the adjustment unit selects data to be transmitted to the external device from the transmission target data in units of regions determined based on a frame, a pixel, or an object. .   The information processing system according to claim 8 or 9, wherein the storage unit stores, in a storage medium, data that is not selected by the adjustment unit among the transmission target data. The adjusting unit selects data to be transmitted to the external device from the transmission target data in units of areas determined based on frames, pixels, or objects,
11. The storage unit stores data that is not selected by the adjustment unit in the transmission target data in a storage medium in the same unit as a unit of data selection by the adjustment unit. The information processing system described.   The information processing system according to claim 1, wherein the adjustment unit compresses the transmission target data.   Data for bringing the transmission target data closer to the original transmission target data is the transmission target data compressed at a compression rate lower than the compression rate when compression is performed by the adjustment unit, or by the adjustment unit. The information processing system according to claim 12, wherein the data is the transmission target data before being compressed. Acquisition means for acquiring data specifying information which is information for specifying data stored by the storage means;
Selecting means for selecting data specified by the data specifying information from the data stored by the storage means;
The information processing system according to claim 1, wherein the transmission unit transmits the data selected by the selection unit.   15. The information processing system according to claim 14, wherein the data specifying information includes time information when the data is obtained or identification information of the data.   The information processing system according to claim 14, wherein the data specifying information includes information for specifying a plurality of data.   The information processing system according to any one of claims 14 to 16, wherein the data specifying information includes information indicating a priority order for the data stored by the storage unit.   The information processing system according to any one of claims 1 to 17, further comprising a determination unit that determines whether or not a data amount of the transmission target data exceeds a threshold value.   The information processing system according to any one of claims 1 to 18, further comprising a plurality of information processing apparatuses each including the adjustment unit, the storage unit, and the transmission unit. The adjustment unit included in the information processing apparatus adjusts the transmission target data to be transmitted to the information processing apparatus that is the next order of the information processing apparatus in a predetermined order,
The transmission means included in the information processing apparatus transmits data to the information processing apparatus that is in the next order of the information processing apparatus in a predetermined order,
The transmission target data adjusted by the adjustment unit included in the information processing apparatus is determined in advance as data including an image of a subject included in a captured image captured by an imaging unit connected to the information processing apparatus. The information processing system according to claim 19, further comprising: data including an image of a subject transmitted from the information processing apparatus that is in the order immediately preceding the information processing apparatus in the order.   21. The information processing system according to claim 19, wherein the plurality of information processing apparatuses are daisy chain connected. Setting means for setting a viewpoint for the subject;
And generating means for generating an image of the subject when viewed from the viewpoint set by the setting means, using images of the subject included in the plurality of captured images taken from a plurality of directions. The information processing system according to any one of claims 1 to 21, wherein the information processing system is characterized in that: An adjustment process for adjusting transmission target data, which is data including an image of a subject included in a captured image, and is data to be transmitted to an external device;
A storage step of storing, in a storage medium, data for bringing the transmission target data adjusted in the adjustment step closer to the original transmission target data;
A transmission step of transmitting data to the external device,
In the transmission step, after transmitting the transmission target data adjusted in the adjustment step, data for transmitting the transmission target data adjusted in the adjustment step to the original transmission target data is transmitted. Information processing method.   A program that causes a computer to function as the information processing system according to any one of claims 1 to 22.

Images