JP2019047431A - Image processing device, control method thereof, and image processing system - Google Patents

Abstract

To make it possible to precisely detect a change in an installation state of an imaging device.SOLUTION: An image processing device includes: reception means for receiving a photographic image taken with an imaging device; separation means for separating a background image from the photographic image; first storage means for storing a reference image based on a first photographic image taken at a first time with the imaging device; second storage means for storing a current image based on a second photographic image taken at a second time; and detection means for detecting a state change of the imaging device by comparing the reference image and current image. The current image is the background image separated from the second photographic image by the separation means.SELECTED DRAWING: Figure 24

Description

  The present invention relates to a technology for detecting a change in the installation state of an imaging device.

  Recently, a technique for generating virtual viewpoint content using a plurality of viewpoint images obtained by installing a plurality of cameras at different positions in a stadium or the like at different positions and capturing images from multiple viewpoints has attracted attention. The virtual viewpoint content allows, for example, a highlight scene of soccer or basketball to be viewed from various angles, so that the user can obtain a higher sense of reality compared to a normal image.

  By the way, at the time of installation of the plurality of cameras for capturing a plurality of viewpoint images described above and the workflow before capturing, calibration at the time of installation is included. By detecting the change in the camera installation state after calibration at the time of installation and performing calibration again, it is possible to prevent the deterioration of the image quality of the virtual viewpoint content using the multi-viewpoint image. Specifically, the image after calibration at the time of installation is used as a reference image, and a change in the camera installation state is detected by detecting a change in the image in comparison with an image (current image) obtained thereafter. Patent Document 1 discloses a technique in which a background image in which no subject is present is prepared in advance, and the difference between the background image and the current captured image is compared to determine the presence or absence of a subject.

Unexamined-Japanese-Patent No. 2004-172946

  However, in the above-described prior art, when the captured image (comparative image) used for comparison includes the foreground of a person or the like that changes with time, the change in the camera installation state may not be accurately detected. . That is, when there are players or people in the stadium in the field of the stadium, it is not possible to take a comparative image necessary for accurate detection. Therefore, for example, it is necessary to stand by until there are no more people in the field and to take a comparison image.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a technology capable of accurately detecting a change in the installation state of an imaging device.

  In order to solve the above-mentioned problems, the image processing apparatus according to the present invention has the following configuration. That is, the image processing apparatus comprises: receiving means for receiving a captured image captured by the imaging device; separating means for separating a background image from the captured image received by the receiving means; and at a first time by the imaging device. First storage means for storing a reference image based on a first captured image captured; and current based on a second captured image captured at a second time different from the first time by the imaging device A second storage unit for storing an image, and a detection unit for detecting a change in the state of the imaging device by comparing the reference image and the current image, the current image being the separation unit Are background images separated from the second captured image.

  According to the present invention, it is possible to provide a technology capable of accurately detecting a change in the installation state of an imaging device.

FIG. 1 is a diagram showing an overall configuration of an image processing system according to a first embodiment. It is a block diagram showing functional composition of a camera adapter. It is a block diagram which shows the function structure of an image processing part. It is a block diagram showing functional composition of a front end server. It is a block diagram which shows the function structure of a database. It is a block diagram showing functional composition of a back end server. It is a block diagram showing functional composition of virtual camera operation UI. It is a figure explaining a virtual camera. It is a block diagram which shows the function structure of an end user terminal. It is a flowchart which shows the whole workflow in 1st Embodiment. It is a detailed flowchart of processing at the time of installation. It is a sequence diagram of calibration processing at the time of installation. It is a detailed flowchart of camera parameter estimation processing. It is a sequence diagram of generation processing of three-dimensional model information. It is a flowchart which shows operation | movement of the camera adapter at the time of receiving a captured image from a camera. It is a flow chart which shows operation of a camera adapter at the time of receiving data from an adjoining camera adapter. It is a figure explaining a gaze point group. It is a figure explaining bypass transmission control. It is a figure explaining bypass control in a camera adapter. It is a figure showing an example of the flow of data between a plurality of camera adapters. It is a flowchart of the output process in a transmission part. It is a figure explaining a background image. 5 is a flowchart of various processes in the image processing unit. It is a flow chart of processing in a gap detection informing part. It is a figure which shows a foreground background isolation | separation process illustratively. It is a flowchart of a process in a gap detection informing part in a 2nd embodiment. It is a flowchart of a process in a gap detection informing part in a 3rd embodiment. It is a flowchart of a process in a gap detection informing part in a 4th embodiment. It is a figure which shows the hardware constitutions of a camera adapter.

  Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. The following embodiment is merely an example and is not intended to limit the scope of the present invention.

First Embodiment
As a first embodiment of the image processing apparatus according to the present invention, a system for installing a plurality of cameras and microphones in a facility such as a stadium or a concert hall and performing photography and sound collection will be described below as an example. .

<Whole system configuration>
FIG. 1 is a diagram showing an overall configuration of an image processing system according to the first embodiment. The image processing system 100 includes sensor systems 110 a to 110 z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

  The controller 300 has a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management, parameter setting control, and the like on the blocks constituting the image processing system 100 through the networks 310a to 310c, 180a, 180b, and 170a to 170y. Here, the network may be Ethernet (registered trademark) IEEE standard compliant GbE (Gigabit Ethernet) or 10 GbE, or may be configured by combining an interconnect Infiniband, industrial Ethernet, or the like. Moreover, it is not limited to these, It may be another type of network.

  First, an operation of transmitting 26 sets of images and sounds of the sensor systems 110a to 110z from the sensor system 110z to the image computing server 200 will be described. In the image processing system 100 of this embodiment, sensor systems 110a to 110z are connected by a daisy chain.

  In the present embodiment, 26 sets of systems from the sensor system 110a to the sensor system 110z are simply distinguished as the sensor system 110 unless otherwise described. Similarly, the devices in each sensor system 110 are described as the microphone 111, the camera 112, the camera platform 113, the external sensor 114, and the camera adapter 120 without distinction unless otherwise described. Although 26 sets are described as the number of sensor systems, this is merely an example, and the number is not limited to this. In the present embodiment, unless otherwise noted, the term "image" is described as including the concepts of both moving and still images. That is, the image processing system 100 of the present embodiment can process both still images and moving images. Further, in the present embodiment, the virtual viewpoint content provided by the image processing system 100 will be described focusing on an example in which a virtual viewpoint image and a virtual viewpoint sound are included, but the present invention is not limited thereto. For example, the audio may not be included in the virtual viewpoint content. Also, for example, the sound included in the virtual viewpoint content may be the sound collected by the microphone closest to the virtual viewpoint. Further, in the present embodiment, although the description of the voice is partially omitted for simplification of the description, basically both the image and the voice are processed.

  Each of the sensor systems 110a to 110z has one camera 112a to 112z. That is, the image processing system 100 has a plurality of cameras for photographing a subject from a plurality of directions. The plurality of sensor systems 110 are connected by a daisy chain. With this connection form, it is possible to reduce the number of connection cables and save labor for wiring work in increasing the resolution of the captured image to 4K, 8K, etc. and increasing the capacity of image data accompanying the increase in frame rate. I will specify it here.

  The present invention is not limited to this, and as a connection form, each sensor system 110a to 110z may be connected to the switching hub 180, and may have a star network configuration for transmitting and receiving data between the sensor systems 110 via the switching hub 180.

  Further, although FIG. 1 shows a configuration in which all the sensor systems 110a to 110z are cascaded to form a daisy chain, the present invention is not limited to this. For example, the plurality of sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy-chained among the divided group units. Then, the camera adapter 120 serving as the end of the division unit may be connected to the switching hub to input an image to the image computing server 200. Such an arrangement is particularly effective in stadiums. For example, there may be a case where a stadium is composed of a plurality of floors and the sensor system 110 is deployed on each floor. In this case, the input to the image computing server 200 can be performed every floor or every half cycle of the stadium, and installation is simplified even in a place where it is difficult to connect all the sensor systems 110 by one daisy chain and System flexibility can be achieved.

  Further, control of image processing in the image computing server 200 can be switched depending on whether there is one or two or more camera adapters 120 connected in a daisy chain and performing image input to the image computing server 200. . That is, the control is switched depending on whether the sensor system 110 is divided into a plurality of groups. When there is one camera adapter 120 that performs image input, images are generated all around the stadium while performing image transmission by daisy chain connection, so the image computing server 200 is synchronized when the image data of all the circumferences are aligned It is taken. That is, if the sensor system 110 is not divided into groups, synchronization can be achieved.

  However, when there are a plurality of camera adapters 120 that perform image input (the sensor system 110 is divided into groups), it is conceivable that the delay may differ depending on the lanes (paths) of the respective daisy chains. Therefore, it is clearly stated that it is necessary to perform image processing in the latter stage while checking the concentration of the image data by synchronous control in which the image computing server 200 waits and synchronizes until the image data of all the circumferences are complete.

  In the present embodiment, the sensor system 110a includes a microphone 111a, a camera 112a, a camera platform 113a, an external sensor 114a, and a camera adapter 120a. The present invention is not limited to this configuration, as long as at least one camera adapter 120a and one camera 112a or one microphone 111a are included. Further, for example, the sensor system 110a may be configured by one camera adapter 120a and a plurality of cameras 112a, or may be configured by a single camera 112a and a plurality of camera adapters 120a. That is, the plurality of cameras 112 and the plurality of camera adapters 120 in the image processing system 100 correspond to each other by N pairs M (N and M are both integers of 1 or more). The sensor system 110 may also include devices other than the microphone 111a, the camera 112a, the camera platform 113a, and the camera adapter 120a. Also, the camera 112 and the camera adapter 120 may be integrally configured. Furthermore, the front end server 230 may have at least a part of the functions of the camera adapter 120. In the present embodiment, the sensor systems 110 b to 110 z are omitted because they have the same configuration as the sensor system 110 a. The configuration is not limited to the same as that of the sensor system 110a, and each sensor system 110 may have a different configuration.

  The sound collected by the microphone 111a and the image captured by the camera 112a are subjected to image processing described later in the camera adapter 120a, and then transmitted to the camera adapter 120b of the sensor system 110b through the daisy chain 170a. . Similarly, the sensor system 110b transmits the collected voice and the captured image to the sensor system 110c along with the image and voice acquired from the sensor system 110a.

  By continuing the operation described above, the images and sounds acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z to the switching hub 180 via the network 180b and then transmitted to the image computing server 200.

  In the present embodiment, the cameras 112a to 112z and the camera adapters 120a to 120z are separated, but may be integrated by the same housing. In that case, the microphones 111 a to 111 z may be incorporated in the integrated camera 112 or may be connected to the outside of the camera 112.

  Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 of the present embodiment processes data acquired from the sensor system 110 z. The image computing server 200 includes a front end server 230, a database (DB) 250, a back end server 270, and a time server 290.

  The time server 290 has a function of distributing time and synchronization signals, and distributes time and synchronization signals to the sensor systems 110a to 110z through the switching hub 180. The camera adapters 120a to 120z having received the time and synchronization signal genlock the cameras 112a to 112z based on the time and synchronization signal to perform image frame synchronization. That is, the time server 290 synchronizes the imaging timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, so that it is possible to suppress the degradation of the quality of the virtual viewpoint image due to the shift of the capturing timing. In the present embodiment, the time server 290 manages time synchronization of a plurality of cameras 112. However, the present invention is not limited to this. Each camera 112 or each camera adapter 120 independently performs processing for time synchronization. May be

  The front end server 230 reconstructs a segmented transmission packet from the image and sound acquired from the sensor system 110 z and converts the data format, and then converts it into the database 250 according to the camera identifier, data type, and frame number. Write.

  Next, the back end server 270 receives specification of a viewpoint from the virtual camera operation UI 330, reads out the corresponding image and audio data from the database 250 based on the received viewpoint, and performs rendering processing to generate a virtual viewpoint image Do.

  The configuration of the image computing server 200 is not limited to this. For example, at least two of the front end server 230, the database 250, and the back end server 270 may be integrally configured. Also, a plurality of at least one of the front end server 230, the database 250, and the back end server 270 may be included. In addition, devices other than the above-described devices may be included at any position in the image computing server 200. Furthermore, the end user terminal 190 or the virtual camera operation UI 330 may have at least a part of the functions of the image computing server 200.

  The image subjected to the rendering process is transmitted from the back end server 270 to the end user terminal 190, and the user operating the end user terminal 190 can view and listen to images according to the designation of the viewpoint. That is, the back-end server 270 generates virtual viewpoint content based on the captured images (multi-viewpoint images) captured by the plurality of cameras 112 and the viewpoint information. More specifically, the back-end server 270 is a virtual viewpoint based on, for example, image data of a predetermined area extracted from images captured by the plurality of cameras 112 by the plurality of camera adapters 120 and a viewpoint designated by the user operation. Generate content. Then, the back end server 270 provides the end user terminal 190 with the generated virtual viewpoint content. Details of extraction of the predetermined area by the camera adapter 120 will be described later.

  The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when the subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the designated viewpoint. The virtual viewpoint (virtual viewpoint) may be designated by the user, or may be automatically designated based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily specified by the user. Further, an image corresponding to a viewpoint specified by the user from a plurality of candidates and an image corresponding to a viewpoint automatically specified by the apparatus are also included in the virtual viewpoint image. In the present embodiment, an example in which audio data (audio data) is included in virtual viewpoint content is mainly described. However, audio data may not necessarily be included. Also, the back end server 270 transmits the virtual viewpoint image to the H.264 system. Alternatively, the data may be compressed and encoded by a standard technology represented by H.264 and HEVC, and then transmitted to the end user terminal 190 using the MPEG-DASH protocol. Also, the virtual viewpoint image may be transmitted to the end user terminal 190 uncompressed. In particular, the former that performs compression coding assumes a smartphone or a tablet as the end user terminal 190, and the latter assumes a display capable of displaying uncompressed images. That is, it is specified that the image format can be switched according to the type of the end user terminal 190. Further, the transmission protocol of the image is not limited to the MPEG-DASH, and for example, HLS (HTTP Live Streaming) or another transmission method may be used.

  Thus, the image processing system 100 has three functional domains: a video acquisition domain, a data storage domain, and a video generation domain. The image acquisition domain includes sensor systems 110a to 110z, the data storage domain includes a database 250, a front end server 230 and a back end server 270, and the image generation domain includes a virtual camera operation UI 330 and an end user terminal 190. Note that the present invention is not limited to this configuration. For example, the virtual camera operation UI 330 can also obtain an image directly from the sensor systems 110a to 110z. However, in the present embodiment, not the method of directly acquiring an image from the sensor systems 110a to 110z but the method of arranging the data storage function in the middle is adopted. Specifically, the front end server 230 converts the image data and audio data generated by the sensor systems 110a to 110z and the meta information of those data into the common schema and data type of the database 250. Thereby, even if the cameras 112 of the sensor systems 110a to 110z change to cameras of other types, the front end server 230 can absorb the changed difference and register it in the database 250. This can reduce the possibility that the virtual camera operation UI 330 does not operate properly when the camera 112 is changed to another model camera.

  Also, the virtual camera operation UI 330 is configured to access via the back end server 270 without directly accessing the database 250. The back-end server 270 performs common processing related to image generation processing, and the difference part of the application related to the operation UI is performed using the virtual camera operation UI 330. As a result, in the development of the virtual camera operation UI 330, it is possible to focus on the development for the function request of the UI operation device and the UI for operating the virtual viewpoint image to be generated. The back-end server 270 can also add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 330. By this, it is possible to flexibly cope with the request of the virtual camera operation UI 330.

  As described above, in the image processing system 100, the virtual viewpoint image is generated by the back end server 270 based on the image data based on the photographing by the plurality of cameras 112 for photographing the subject from a plurality of directions. The image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be logically configured.

<Functional configuration of each node>
Next, the functional configuration of each node (camera adapter 120, front end server 230, database 250, back end server 270, virtual camera operation UI 330, end user terminal 190) constituting the system will be described.

  FIG. 2 is a block diagram showing a functional configuration of the camera adapter 120. As shown in FIG. The details of the flow of data between the functional blocks of the camera adapter 120 will be described later with reference to FIG.

  The camera adapter 120 includes a network adapter 6110, a transmission unit 6120, an image processing unit 6130, and an external device control unit 6140. The network adapter 6110 includes a data transmission / reception unit 6111 and a time control unit 6112.

  The data transmission / reception unit 6111 performs data communication with the other camera adapter 120, the front end server 230, the time server 290, and the control station 310 via the daisy chain 170, the network 291, and the network 310a. For example, the data transmission / reception unit 6111 outputs the foreground image and the background image separated by the foreground / background separation unit 6131 from the image captured by the camera 112 to another camera adapter 120. The camera adapter 120 of the output destination is the next camera adapter 120 among the camera adapters 120 in the image processing system 100 in the order determined in advance according to the processing of the data routing processing unit 6122. Each camera adapter 120 outputs a foreground image and a background image as an image of a predetermined format, so that a virtual viewpoint image is generated based on the foreground image and the background image captured from a plurality of viewpoints. Note that there may be a camera adapter 120 that outputs a foreground image separated from a captured image and does not output a background image.

  The time control unit 6112 performs, for example, a function of storing a time stamp of data transmitted to and received from the time server 290 and time synchronization with the time server 290 in compliance with the Ordinay Clock of IEEE 1588 standard. The present invention is not limited to the IEEE 1588, and time synchronization with the time server 290 may be realized by another EtherAVB standard or a unique protocol. In the present embodiment, a NIC (Network Interface Card) is used as the network adapter 6110. However, the present invention is not limited to the NIC, and the same other interface may be used. Also, IEEE 1588 is updated as a standard such as IEEE 1588-2002 and IEEE 1588-2008, and the latter is also called PTPv2 (Precision Time Protocol Version 2).

  The transmission unit 6120 has a function of controlling data transmission to the switching hub 180 and the like via the network adapter 6110, and is configured by the following function units.

  A data compression / decompression unit 6121 has a function of performing compression to which data is transmitted and received via the data transmission / reception unit 6111 by applying a predetermined compression method, compression ratio, and frame rate, and a function of decompressing compressed data. have.

  The data routing processing unit 6122 determines the routing destination of the data received by the data transmitting / receiving unit 6111 and the data processed by the image processing unit 6130 using the data held by the data routing information holding unit 6125 described later. Furthermore, it has a function of transmitting data to the determined routing destination. As a routing destination, it is preferable to use a camera adapter 120 corresponding to the cameras 112 focused on the same fixation point in order to perform image processing because the image frame correlation between the cameras 112 is high. In accordance with the determination by the data routing processing unit 6122 of each of the plurality of camera adapters 120, the order of the camera adapters 120 for outputting the foreground image and the background image in a relay format in the image processing system 100 is determined.

  The time synchronization control unit 6123 conforms to PTP (Precision Time Protocol) of the IEEE 1588 standard, and has a function of performing processing relating to time synchronization with the time server 290. Note that time synchronization may be performed using another similar protocol instead of PTP.

  The image / sound transmission processing unit 6124 has a function of creating a message for transferring image data or sound data to another camera adapter 120 or the front end server 230 via the data transmission / reception unit 6111. The message includes image data or audio data, and meta information of each data. The meta information of the present embodiment includes a time code or sequence number when shooting an image or sampling an audio, a data type, an identifier indicating an individual of the camera 112 or the microphone 111, and the like. The image data or audio data to be transmitted may be compressed by the data compression / decompression unit 6121. Also, the image / voice transmission processing unit 6124 receives a message from another camera adapter 120 via the data transmission / reception unit 6111. Then, in accordance with the data type included in the message, the data information fragmented in the packet size specified in the transmission protocol is restored into image data or audio data. If the data is compressed when the data is restored, the data compression / decompression unit 6121 performs decompression processing.

  The data routing information holding unit 6125 has a function of holding address information for determining a transmission destination of data transmitted and received by the data transmission / reception unit 6111. The routing method will be described later.

  The image processing unit 6130 has a function of processing image data captured by the camera 112 under control of the camera control unit 6141 and image data received from other camera adapters 120, and includes the following functional units: There is.

  The foreground / background separation unit 6131 has a function of separating image data captured by the camera 112 into a foreground image and a background image. That is, the foreground / background separation unit 6131 of each of the plurality of camera adapters 120 extracts a predetermined area from an image captured by the corresponding camera 112 among the plurality of cameras 112. The predetermined area is, for example, a foreground image obtained as a result of object detection on the photographed image, and the foreground / background separation unit 6131 separates the photographed image into a foreground image and a background image by this extraction. The object is, for example, a person. However, the object may be a specific person (such as a player, a director, and / or an umpire), or an object such as a ball or a goal, for which an image pattern is predetermined. Also, a moving body may be detected as an object. An image of a portion corresponding to the above object of the virtual viewpoint image generated in the image processing system 100 by separating and processing the foreground image including important objects such as a person and the background region not including such objects Can improve the quality of In addition, by separating the foreground and the background from each other by the plurality of camera adapters 120, it is possible to disperse the load in the image processing system 100 provided with the plurality of cameras 112. The predetermined area is not limited to the foreground image, and may be, for example, a background image. Further, the process of separating into the foreground image and the background image performed in the foreground / background separation unit 6131 for the detection of deviation described later may be the above-mentioned image processing. However, in the present embodiment, foreground / background separation processing is performed to separate the foreground and the background using both temporal correlation and spatial correlation for more accurate measurement of deviation. The foreground / background separation used for this shift detection will be described later.

  The three-dimensional model information generation unit 6132 uses the foreground image separated by the foreground / background separation unit 6131 and the foreground image received from the other camera adapter 120, and uses, for example, the principle of a stereo camera to generate image information related to a three-dimensional model. Have the ability to generate

  The calibration control unit 6133 has a function of acquiring image data necessary for calibration from the camera 112 via the camera control unit 6141 and transmitting the image data to the front end server 230 that performs calculation processing related to calibration. . In the present embodiment, the front end server 230 performs calculation processing related to calibration, but the node performing the calculation processing is not limited to the front end server 230. For example, the computing process may be performed at another node such as the control station 310 or the camera adapter 120 (including the other camera adapter 120). Also, the calibration control unit 6133 has a function of performing calibration (dynamic calibration) during shooting according to a preset parameter on image data acquired from the camera 112 via the camera control unit 6141. doing.

  The shift detection / notification unit 6134 has a function of detecting displacement (hereinafter referred to as shift) of an image from the background image separated by the foreground / background separation unit 6131. First, temporal correlation is performed on an image captured at the time of camera initialization, and foreground / background separation processing is performed to separate foreground and background from spatial correlation. Then, the obtained background image is stored as a reference image. Next, the current image is photographed at startup or periodically, the same foreground / background separation processing is performed, and the obtained background image is stored as the current image. Then, the reference image and the current image are compared to detect a deviation. If the deviation is larger than a predetermined reference value, it is determined that the installation state of the camera has changed, and an alarm is notified.

  The external device control unit 6140 has a function of controlling devices connected to the camera adapter 120, and is configured of the following functional blocks.

  The camera control unit 6141 is connected to the camera 112, and has a function of controlling the camera 112, acquiring a captured image, providing a synchronization signal, setting a time, and the like. For control of the camera 112, for example, setting and reference of shooting parameters (number of pixels, color depth, frame rate, setting of white balance, etc.), status of the camera 112 (during shooting, stopping, synchronizing, error etc.) Acquisition, start and stop of shooting, and focus adjustment. In the present embodiment, focus adjustment is performed via the camera 112. However, when a removable lens is attached to the camera 112, the camera adapter 120 is connected to the lens to directly adjust the lens. It is also good. In addition, the camera adapter 120 may perform lens adjustment such as zooming via the camera 112. The synchronization signal is provided by the time synchronization control unit 6123 using the time synchronized with the time server 290 and providing the camera 112 with a photographing timing (control clock). The time setting is performed by providing the time synchronized with the time server 290 by the time synchronization control unit 6123 with, for example, a time code conforming to the format of SMPTE 12M. As a result, the time code provided to the image data received from the camera 112 is given. The format of the time code is not limited to SMPTE 12M, and may be another format. In addition, the camera control unit 6141 may not provide a time code to the camera 112, and may assign a time code to image data received from the camera 112.

  The microphone control unit 6142 is connected to the microphone 111, and has a function of controlling the microphone 111, starting and stopping sound collection, and acquiring voice data collected. The control of the microphone 111 is, for example, gain adjustment, state acquisition, or the like. Further, similarly to the camera control unit 6141, timing and time code for audio sampling are provided to the microphone 111. As clock information for audio sampling timing, time information from the time server 290 is converted to, for example, a 48 KHz word clock and supplied to the microphone 111.

  The pan head control unit 6143 is connected to the pan head 113 and has a function of controlling the pan head 113. The control of the pan head 113 includes, for example, pan / tilt control, state acquisition, and the like.

  The sensor control unit 6144 is connected to the external sensor 114, and has a function of acquiring sensor information sensed by the external sensor 114. For example, when a gyro sensor is used as the external sensor 114, information representing vibration can be acquired. Then, using the vibration information acquired by the sensor control unit 6144, the image processing unit 6130 can generate an image in which the vibration is suppressed prior to the processing in the foreground / background separation unit 6131. The vibration information is used, for example, in the case where image data of an 8K camera is cut out in a size smaller than the original 8K size in consideration of the vibration information, and alignment with the image of the camera 112 installed adjacently is performed. . Thereby, even if the casing vibration of the building propagates to each camera at different frequencies, alignment is performed using this function provided in the camera adapter 120. As a result, it is possible to generate image data that has been subjected to electronic image stabilization, and it is possible to obtain an effect of reducing the processing load of alignment for the number of cameras 112 in the image computing server 200. In addition, the sensor of the sensor system 110 is not necessarily limited to the external sensor 114, and even if it is a sensor incorporated in the camera adapter 120, the same effect is acquired.

  Subsequently, the hardware configuration of each device constituting each of the above-described embodiments will be described in more detail. As described above, the camera adapter 120 implements hardware such as an FPGA and / or an ASIC, and the hardware executes the above-described processes. The same applies to various devices in the sensor system 110, the front end server 230, the database 250, the back end server 270, and the controller 300. However, at least one of the above-described devices may execute the processing of each embodiment by software processing using, for example, a CPU, a GPU, a DSP, or the like.

  FIG. 29 is a diagram showing a hardware configuration of the camera adapter 120. As shown in FIG. Here, it is assumed that the functional configuration shown in FIG. 2 is realized by software processing using, for example, a CPU, a GPU, a DSP, and the like. Note that devices such as the front end server 230, the database 250, the back end server 270, the control station 310, the virtual camera operation UI 330, and the end user terminal 190 can also have the hardware configuration shown in FIG. The camera adapter 120 includes a CPU 1201, a ROM 1202, a RAM 1203, an auxiliary storage device 1204, a display unit 1205, an operation unit 1206, a communication unit 1207, and a bus 1208.

  The CPU 1201 controls the entire camera adapter 120 using computer programs and data stored in the ROM 1202 and the RAM 1203. The ROM 1202 stores programs and parameters that do not need to be changed. The RAM 1203 temporarily stores programs and data supplied from the auxiliary storage device 1204, data supplied from the outside via the communication unit 1207, and the like. The auxiliary storage device 1204 is configured by, for example, a hard disk drive and stores content data such as still images and moving images.

  The display unit 1205 is configured of, for example, a liquid crystal display, and displays a GUI (Graphical User Interface) or the like for the user to operate the camera adapter 120. The operation unit 1206 includes, for example, a keyboard and a mouse, and receives various operations from the user and inputs various instructions to the CPU 1201. A communication unit 1207 communicates with external devices such as the camera 112 and the front end server 230. For example, when the camera adapter 120 is connected to an external device by wire, a LAN cable or the like is connected to the communication unit 1207. When the camera adapter 120 has a function of performing wireless communication with an external device, the communication unit 1207 includes an antenna. A bus 1208 connects each unit of the camera adapter 120 to transmit information.

  For example, a part of the processing of the camera adapter 120 may be performed by an FPGA, and another part of the processing may be realized by software processing using a CPU. Further, each component of the camera adapter 120 shown in FIG. 29 may be configured by a single electronic circuit or may be configured by a plurality of electronic circuits. For example, the camera adapter 120 may include a plurality of electronic circuits operating as the CPU 1201. The processing speed of the camera adapter can be improved by performing processing of the CPU 1201 in parallel by the plurality of electronic circuits.

  In each of the embodiments described above, the display unit 1205 and the operation unit 1206 exist inside the camera adapter 120, but the camera adapter 120 may not include at least one of the display unit 1205 and the operation unit 1206. In addition, at least one of the display unit 1205 and the operation unit 1206 exists as another device outside the camera adapter 120, and the CPU 1201 controls the display control unit that controls the display unit 1205, and operation control that controls the operation unit 1206. It may operate as a part. The same applies to other devices in the image processing system 100. For example, the front end server 230, the database 250, and the back end server 270 may not include the display unit 1205, and the control station 310, the virtual camera operation UI 330, and the end user terminal 190 may include the display unit 1205. Moreover, in the above-mentioned each embodiment, it demonstrated centering on the example in case the image processing system 100 is installed in institutions, such as a stadium and a concert hall. Other examples of the facility include, for example, an amusement park, a park, a racetrack, a bicycle race track, a casino, a pool, a skating rink, a ski resort, a live house and the like. In addition, events performed in various facilities may be performed indoors or outdoor. Moreover, the facility in each embodiment includes a facility that is temporarily (limitedly) built.

  FIG. 3 is a block diagram showing the functional configuration of the image processing unit 6130 in the camera adapter 120. As shown in FIG. The calibration control unit 6133 performs color correction processing for suppressing variation in color for each camera on the input image, and shake correction for stabilizing the position of the image against shake due to camera vibration. Perform processing (electronic anti-vibration processing) and the like.

  The functional blocks of the foreground / background separator 6131 will be described. The foreground separation unit 5001 performs separation processing of the foreground image on the image data on which the alignment of the image of the camera 112 has been performed, by comparison with the background image 5002.

  The background updating unit 5003 generates a new background image using the image in which the background image 5002 and the camera 112 are aligned, and updates the background image 5002 with the new background image.

  The background cutting out unit 5004 controls to cut out a part of the background image 5002. Here, the function of the three-dimensional model information generation unit 6132 will be described.

  The three-dimensional model processing unit 5005 uses a foreground image separated by the foreground separation unit 5001 and a foreground image of another camera 112 received via the transmission unit 6120, for example, based on the principle of a stereo camera etc. It sequentially generates related image information.

  The other camera foreground reception unit 5006 receives the foreground image whose foreground / background is separated by the other camera adapter 120.

  The camera parameter receiving unit 5007 receives internal parameters specific to the camera (focal length, image center, lens distortion parameter, etc.) and external parameters (rotation matrix, position vector, etc.) representing the position and orientation of the camera. These parameters are information obtained by the calibration processing described later, and are transmitted and set from the control station 310 to the target camera adapter 120. Next, the three-dimensional model processing unit 5005 generates three-dimensional model information by the camera parameter receiving unit 5007 and the other camera foreground receiving unit 5006.

  Here, the function of the deviation detection notification unit 6134 will be described. The shift detection notification unit 6134 has a function of detecting a shift of an image from the background image separated by the foreground / background separation unit 6131.

  First, an image is captured after the calibration at the time of initial setting of the camera (first time). Next, the foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the temporal correlation and the spatial correlation with the image captured at the time of this camera initialization. Details regarding this temporal correlation and foreground / background separation processing for separating foreground and background from spatial correlation will be described later. Then, the background image 5002 is stored as a reference image in the reference background image storage unit 5008 (first storage unit). Next, the current image is taken at startup or periodically (second time). Next, with respect to this image, the foreground separation unit 5001 performs the same temporal correlation as described above and foreground / background separation processing for separating the foreground and the background from the spatial correlation. Then, the background image 5002 is stored in the comparison current background image storage unit 5009 (second storage unit) as a comparison current image (comparison current image). Next, the shift detection unit 5010 compares the reference image of the reference background image storage unit 5008 with the comparison current image of the comparison current background image storage unit 5009 to detect a shift. The comparison method uses a method using vectors between corresponding feature points, and other known techniques. If the deviation is larger than a predetermined reference value, it is determined that the installation state of the camera has changed, and an alarm is notified and alarm information is transmitted to the transmission unit 6120.

  FIG. 4 is a block diagram showing a functional configuration of the front end server 230. As shown in FIG. The control unit 2110 is configured by a CPU, a DRAM, a storage medium such as an HDD or a NAND memory storing program data and various data, and hardware such as Ethernet (registered trademark). Then, each functional block of the front end server 230 and the entire system of the front end server 230 are controlled. In addition, mode control is performed to switch operation modes such as a calibration operation, a preparation operation before shooting, and an operation during shooting. Further, the control instruction from the control station 310 is received through Ethernet (registered trademark), and switching of each mode, data input / output, etc. are performed. Similarly, stadium CAD data (stadium shape data) is acquired from the control station 310 through the network, and stadium CAD data is transmitted to the CAD data storage unit 2135 and the photographed data file generation unit 2180. The stadium CAD data (stadium shape data) in the present embodiment is three-dimensional data indicating the shape of the stadium, and may be a mesh model or data representing another three-dimensional shape, and is not limited to the CAD format.

  The data input control unit 2120 is network-connected to the camera adapter 120 via a communication path such as Ethernet (registered trademark) and the switching hub 180. Then, the data input control unit 2120 acquires the foreground image, the background image, the three-dimensional model of the subject, the sound data, and the camera calibration captured image data from the camera adapter 120 through the network. Here, the foreground image is image data based on the foreground area of the captured image for generation of a virtual viewpoint image, and the background image is image data based on the background area of the captured image. The camera adapter 120 specifies a foreground area and a background area according to the result of detection processing of a predetermined object on a captured image by the camera 112, and generates a foreground image and a background image. The predetermined object is, for example, a person. The predetermined object may be a specific person (such as a player, a manager, and / or an umpire). Further, the predetermined object may include an object such as a ball or a goal, for which an image pattern is predetermined. In addition, a moving object may be detected as a predetermined object.

  Further, the data input control unit 2120 transmits the acquired foreground image and background image to the data synchronization unit 2130, and transmits camera calibration photographed image data to the calibration unit 2140. Also, the data input control unit 2120 has a function of performing compression / decompression of received data, data routing processing, and the like. Although both the control unit 2110 and the data input control unit 2120 have a communication function by a network such as Ethernet (registered trademark), the communication function may be shared by these. In that case, a method may be used in which the data input control unit 2120 receives an instruction by a control command from the control station 310 or stadium CAD data and sends it to the control unit 2110.

  The data synchronization unit 2130 temporarily stores the data acquired from the camera adapter 120 on the DRAM, and buffers the foreground image, the background image, the audio data, and the three-dimensional model data until they are aligned. The foreground image, the background image, the sound data, and the three-dimensional model data are collectively referred to as shooting data hereinafter. The shooting data is attached with meta information such as routing information, time code information (time information), camera identifier and the like, and the data synchronization unit 2130 confirms the attribute of the data based on the meta information. Thus, the data synchronization unit 2130 determines that the data is identical by determining that the data is the same at the same time. This is because, for the data transferred from each camera adapter 120 by the network, the reception order of the network packets is not guaranteed, and it is necessary to buffer until the data necessary for file generation is complete.

  When the data is prepared, the data synchronization unit 2130 transmits the foreground image and the background image to the image processing unit 2150, the three-dimensional model data to the three-dimensional model combination unit 2160, and the audio data to the photographed data file generation unit 2180. Note that the data to be aligned here is data necessary for file generation in a shooting data file generation unit 2180 described later. Also, the background image may be taken at a different frame rate than the foreground image. For example, when the frame rate of the background image is 1 fps, one background image is acquired every one second, and therefore, it may be assumed that all the data are prepared without the background image for the time when the background image is not acquired. . Further, the data synchronization unit 2130 notifies the database 250 of information indicating that data aggregation can not be performed when the predetermined time has elapsed and the data is not complete. Then, when storing the data, the database 250 in the latter stage stores information indicating a missing data together with the camera number and the frame number. This makes it possible to automatically notify before rendering whether or not a desired image can be formed from the captured image of the camera 112 in accordance with the viewpoint instruction from the virtual camera operation UI 330 to the back end server 270. . As a result, the visual load on the operator of the virtual camera operation UI 330 can be reduced.

  The CAD data storage unit 2135 stores three-dimensional data indicating the stadium shape received from the control unit 2110 in a storage medium such as a DRAM, an HDD, or a NAND memory. Then, the stadium shape data stored when the stadium shape data request is received is transmitted to the image combining unit 2170.

  The calibration unit 2140 performs a calibration operation of the camera, and sends camera parameters obtained by the calibration to a non-shooting data file generation unit 2185 described later. At the same time, camera parameters are held in its own storage area, and camera parameter information is provided to a three-dimensional model combining unit 2160 described later.

  An image processing unit 2150 performs processing such as matching of colors and luminance values between cameras with a foreground image or background image, development processing when RAW image data is input, and correction of lens distortion of the camera. . Then, the foreground image subjected to the image processing is transmitted to the photographed data file generation unit 2180, and the background image is transmitted to 2170.

  The three-dimensional model combining unit 2160 combines the three-dimensional model data of the same time acquired from the camera adapter 120 using the camera parameters generated by the calibration unit 2140. Then, using a method called Visual Hull, three-dimensional model data of the foreground image in the entire stadium is generated. The generated three-dimensional model is transmitted to the imaging data file generation unit 2180.

  The image combining unit 2170 obtains a background image from the image processing unit 2150, obtains three-dimensional shape data (stadium shape data) of a stadium from the CAD data storage unit 2135, and obtains a background image with respect to coordinates of the obtained three-dimensional shape data of the stadium. Identify the location of When the position with respect to the coordinates of the three-dimensional shape data of the stadium can be specified for each of the background images, the background images are combined into one background image. The back-end server 270 may perform the creation of three-dimensional shape data of the present background image.

  The photographed data file generation unit 2180 combines voice data from the data synchronization unit 2130, a foreground image from the image processing unit 2150, three-dimensional model data from the three-dimensional model combining unit 2160, and a three-dimensional shape from the image combining unit 2170. Get background image. Then, the acquired data is output to the DB access control unit 2190. Here, the shooting data file generation unit 2180 associates and outputs these data based on the respective time information. However, part of these data may be associated and output. For example, the shooting data file generation unit 2180 associates and outputs the foreground image and the background image based on the time information of the foreground image and the time information of the background image. Also, for example, the shooting data file generation unit 2180 associates the foreground image, the background image, and the three-dimensional model data with the time information of the foreground image, the time information of the background image, and the time information of the three-dimensional model data. Output. Note that the shooting data file generation unit 2180 may file and output the correlated data according to the type of data, or may group and output a plurality of types of data at every time indicated by the time information. It is also good. By outputting the imaging data associated in this way to the database 250 by the DB access control unit 2190, the back end server 270 can generate a virtual viewpoint image from the foreground image and the background image to which the time information corresponds.

  When the frame rates of the foreground image and the background image acquired by the data input control unit 2120 are different, it is difficult for the shooting data file generation unit 2180 to always associate the foreground image and the background image at the same time and output. Therefore, the shooting data file generation unit 2180 associates the time information of the foreground image with the background image having the time information in a relationship based on a predetermined rule, and outputs it. Here, the background image having time information in a relationship based on the time information of the foreground image and the predetermined rule is, for example, time information closest to the time information of the foreground image among the background images acquired by the imaging data file generation unit 2180 Background image.

  As described above, by associating the foreground image with the background image based on a predetermined rule, even when the frame rates of the foreground image and the background image are different, a virtual viewpoint image is generated from the foreground image and the background image captured at a near time. Can be generated. The method of associating the foreground image with the background image is not limited to the above. For example, a background image having temporal information having a relationship based on predetermined time and temporal information of a foreground image is an acquired background image, and among background images having temporal information corresponding to a time before the foreground image, It may be a background image having temporal information closest to temporal information of the foreground image. According to this method, it is possible to output the associated foreground image and background image with low delay, without waiting for acquisition of a background image having a frame rate lower than that of the foreground image. In addition, a background image having time information in a relationship based on predetermined time rules with time information on a foreground image is an acquired background image, and among background images having time information corresponding to a time later than the foreground image, It may be a background image having temporal information closest to temporal information of the foreground image.

  The non-shooting data file generation unit 2185 acquires camera parameters from the calibration unit 2140 and three-dimensional shape data of the stadium from the control unit 2110, and forms the data according to the file format, and transmits the data to the DB access control unit 2190. Note that camera parameters or stadium shape data, which are data input to the non-shooting data file generation unit 2185, are individually formed according to the file format. That is, when one of the data is received, the non-shooting data file generation unit 2185 individually transmits them to the DB access control unit 2190.

  The DB access control unit 2190 is connected to the database 250 so that high-speed communication can be performed by InfiniBand or the like. Then, the file received from the photographed data file generation unit 2180 and the non-photographed data file generation unit 2185 is transmitted to the database 250. In this embodiment, the shooting data associated with the shooting data file generation unit 2180 based on the time information is transferred to the database 250, which is a storage device connected to the front end server 230 via the network, via the DB access control unit 2190. Output. However, the output destination of the associated imaging data is not limited to this. For example, the front end server 230 outputs shooting data associated based on time information to the back end server 270, which is an image generation device that is connected to the front end server 230 via a network and generates a virtual viewpoint image. May be Also, it may be output to both the database 250 and the back end server 270.

  Further, in the present embodiment, the front end server 230 associates the foreground image with the background image. However, the present invention is not limited to this, and the database 250 may perform the association. For example, database 250 may obtain foreground and background images with temporal information from front end server 230. Then, the database 250 may associate the foreground image with the background image based on the time information of the foreground image and the time information of the background image, and may output the same to the storage unit included in the database 250.

  FIG. 5 is a block diagram showing a functional configuration of the database 250. As shown in FIG. The control unit 2410 is configured by a CPU, a DRAM, a storage medium such as an HDD or a NAND memory storing program data and various data, and hardware such as Ethernet (registered trademark). Then, control of each functional block of the database 250 and the entire system of the database 250 is performed.

  The data input unit 2420 receives a file of shooting data or non-shooting data from the front end server 230 by high-speed communication such as InfiniBand. The received file is sent to the cache 2440. Also, it reads meta information of the received shooting data, and creates a database table to enable access to the acquired data based on information such as time code information, routing information, and camera identifier recorded in the meta information. Do.

  The data output unit 2430 determines whether the data requested from the back end server 270 is stored in a cache 2440, a primary storage 2450, or a secondary storage 2460 described later. Then, by high-speed communication such as InfiniBand, data is read from the stored destination and transmitted to the back end server 270.

  The cache 2440 has a storage device such as a DRAM that can realize high-speed input / output throughput, and stores the imaging data and non-imaging data acquired from the data input unit 2420 in the storage device. The stored data is held in a fixed amount, and when more data is input, the old data is written to the primary storage 2450 at any time, and the written data is overwritten by the new data. Here, the data stored in the cache 2440 by a fixed amount is shooting data for at least one frame. This makes it possible to minimize the throughput in the database 250 and render the latest image frame continuously with low delay when rendering the image in the back end server 270. Here, in order to achieve the above-mentioned purpose, it is necessary to include a background image in cached data. Therefore, when shooting data of a frame having no background image is cached, the background image on the cache is not updated, and is held on the cache as it is. The capacity of the cacheable DRAM is determined by a cache frame size set in advance in the system or an instruction from the control station 310. The non-shooting data is copied to the primary storage 2450 immediately because the frequency of input / output is low and high throughput is not required before the game. The cached data is read by the data output unit 2430.

  The primary storage 2450 is configured by connecting storage media such as SSDs in parallel, etc., and high speed operation can be realized such that writing of a large amount of data from the data input unit 2420 and data reading from the data output unit 2430 can be realized simultaneously. Then, the primary storage 2450 is written out in order from the oldest data stored on the cache 2440. The secondary storage 2460 is composed of an HDD, a tape medium, etc., and a large capacity is more important than high speed, and it is required to be a medium which is inexpensive and suitable for long-term storage as compared with the primary storage. Data stored in the primary storage 2450 is written to the secondary storage 2460 as a backup of the data after the completion of shooting.

  FIG. 6 is a block diagram showing the functional configuration of the back end server 270. As shown in FIG. The back end server 270 includes a data receiving unit 3001, a background texture pasting unit 3002, a foreground texture determining unit 3003, a texture boundary color matching unit 3004, a virtual viewpoint foreground image generating unit 3005, and a rendering unit 3006. Furthermore, it has a virtual viewpoint voice generation unit 3007, a synthesis unit 3008, an image output unit 3009, a foreground object determination unit 3010, a request list generation unit 3011, a request data output unit 3012, and a rendering mode management unit 3014.

  The data receiving unit 3001 receives data transmitted from the database 250 and the controller 300. Also, from the database 250, three-dimensional data (stadium shape data) indicating the shape of a stadium, a foreground image, a background image, a three-dimensional model of the foreground image (hereinafter referred to as a foreground three-dimensional model), and voice are received.

  Also, the data receiving unit 3001 receives virtual camera parameters output from the controller 300 that specifies a viewpoint related to generation of a virtual viewpoint image. The virtual camera parameter is data representing the position, orientation, and the like of the virtual viewpoint, and, for example, a matrix of external parameters and a matrix of internal parameters are used.

  The data acquired by the data reception unit 3001 from the controller 300 is not limited to virtual camera parameters. For example, the information output from the controller 300 includes at least one of a method of specifying a viewpoint, information specifying an application operated by the controller, identification information of the controller 300, and identification information of a user who uses the controller 300. You may Also, the data receiving unit 3001 may obtain, from the end user terminal 190, information similar to the above information output from the controller 300. Furthermore, the data reception unit 3001 may acquire information on the plurality of cameras 112 from an external device such as the database 250 or the controller 300. The information on the plurality of cameras 112 is, for example, information on the number of the plurality of cameras 112 or information on the operation state of the plurality of cameras 112. The operation state of the camera 112 includes, for example, at least one of a normal state, a failure state, a standby state, a start preparation state, and a restart state of the camera 112. The background texture pasting unit 3002 pastes the background mesh model with texture by pasting the background image as a texture to the three-dimensional space shape indicated by the background mesh model (stadium shape data) acquired from the background mesh model management unit 3013. Generate The mesh model is, for example, data representing a three-dimensional space shape such as CAD data as a set of faces. A texture is an image pasted to express the texture of the surface of an object.

  The foreground texture determination unit 3003 determines texture information of the foreground three-dimensional model from the foreground image and the foreground three-dimensional model group.

  The foreground texture boundary color matching unit 3004 performs color matching of texture boundaries from the texture information of each foreground three-dimensional model and each three-dimensional model group, and generates a colored foreground three-dimensional model group for each foreground object.

  The virtual viewpoint foreground image generation unit 3005 perspective-transforms the foreground image group so as to look like a virtual viewpoint based on the virtual camera parameters. The rendering unit 3006 renders a background image and a foreground image to generate a virtual viewpoint image of a full view based on the generation method used for generating a virtual viewpoint image determined by the rendering mode management unit 3014.

  In this embodiment, two rendering modes, model-based rendering (MBR) and image-based rendering (IBR), are used as a virtual viewpoint image generation method.

  The MBR is a method of generating a virtual viewpoint image using a three-dimensional model generated based on a plurality of photographed images obtained by photographing a subject from a plurality of directions. Specifically, using the three-dimensional shape (model) of the target scene obtained by the three-dimensional shape restoration method such as the visual volume intersection method or Multi-View-Stereo (MVS), the appearance of the scene from the virtual viewpoint is It is a technology to generate as an image.

  The IBR is a technology for generating a virtual viewpoint image in which the appearance from a virtual viewpoint is reproduced by transforming and combining an input image group obtained by photographing a target scene from a plurality of viewpoints. In the present embodiment, in the case of using IBR, a virtual viewpoint image is generated based on one or a plurality of photographed images smaller than the plurality of photographed images for generating a three-dimensional model using MBR.

  When the rendering mode is MBR, a panoramic model is generated by combining the background mesh model and the foreground three-dimensional model group generated by the foreground texture boundary color matching unit 3004, and a virtual viewpoint image is generated from the panoramic model.

  When the rendering mode is IBR, a background image viewed from a virtual viewpoint is generated based on the background texture model, and a virtual viewpoint image is generated by combining the foreground image generated by the virtual viewpoint foreground image generation unit 3005 with this. Ru.

  The rendering unit 3006 may use a rendering method other than MBR and IBR. Further, the method of generating a virtual viewpoint image determined by the rendering mode management unit 3014 is not limited to the method of rendering, and the rendering mode management unit 3014 may determine a method of processing other than rendering for generating a virtual viewpoint image. . The rendering mode management unit 3014 determines a rendering mode as a generation method used to generate a virtual viewpoint image, and holds the determination result.

  In the present embodiment, the rendering mode management unit 3014 determines a rendering mode to be used from a plurality of rendering modes. This determination is performed based on the information acquired by the data receiving unit 3001. For example, when the number of cameras specified from the acquired information is equal to or less than a threshold, the rendering mode management unit 3014 determines, as IBR, a generation method used to generate a virtual viewpoint image. On the other hand, when the number of cameras is larger than the threshold, the generation method is determined to be MBR. As a result, when the number of cameras is large, the specifiable range of viewpoints is widened by generating the virtual viewpoint image using the MBR. Further, when the number of cameras is small, it is possible to avoid the image quality degradation of the virtual viewpoint image due to the degradation of the accuracy of the three-dimensional model when using the MBR by using the IBR. Also, for example, the generation method may be determined based on the length of an allowable processing delay time from imaging to image output. Even if the delay time is long, the MBR is used if priority is given to the degree of freedom of the viewpoint, and the IBR is used to require the delay time to be short. Further, for example, when the data receiving unit 3001 acquires information indicating that the controller 300 or the end user terminal 190 can specify the height of the viewpoint, the generation method used for generating the virtual viewpoint image is determined to be MBR. Do. This makes it possible to prevent the user's request for changing the height of the viewpoint from becoming unacceptable due to the generation method being IBR. As described above, by determining the generation method of the virtual viewpoint image according to the situation, the virtual viewpoint image can be generated by the generation method appropriately determined. In addition, by making the plurality of rendering modes switchable in response to a request, the system can be flexibly configured, and can be applied to subjects other than the stadium.

  The rendering mode held by the rendering mode management unit 3014 may be a system preset in the system. Also, the user operating the virtual camera operation UI 330 or the end user terminal 190 may be able to arbitrarily set the rendering mode.

  The virtual viewpoint audio generation unit 3007 generates audio (audio group) that can be heard from the virtual viewpoint based on the virtual camera parameters. The synthesizing unit 3008 synthesizes the image group generated by the rendering unit 3006 and the voice generated by the virtual viewpoint sound generating unit 3007 to generate virtual viewpoint content.

  The image output unit 3009 outputs virtual viewpoint content to the controller 300 and the end user terminal 190 using Ethernet (registered trademark). However, the transmission means to the outside is not limited to Ethernet (registered trademark), and signal transmission means such as SDI, DisplayPort, and HDMI (registered trademark) may be used. The back-end server 270 may output a virtual viewpoint image generated by the rendering unit 3006 and not including audio.

  The foreground object determination unit 3010 determines the displayed foreground object group from the position information of the foreground object indicating the position in space of the foreground object included in the virtual camera parameter and the foreground three-dimensional model, and outputs the foreground object list Do. That is, the foreground object determination unit 3010 performs a process of mapping the image information of the virtual viewpoint to the physical camera 112. The present virtual viewpoint has different mapping results depending on the rendering mode determined by the rendering mode management unit 3014. Therefore, it is specified that the control unit that determines a plurality of foreground objects is deployed to the foreground object determination unit 3010 and performs control in conjunction with the rendering mode.

  The request list generation unit 3011 generates a request list for requesting, from the database 250, a foreground image group and a foreground three-dimensional model group corresponding to a foreground object list of a designated time, and a background image and audio data. For foreground objects, data selected in consideration of virtual viewpoints is required in the database 250, while for background images and audio data, all data for that frame is required. After the back-end server 270 is launched, a request list of background mesh models is generated until a background mesh model is obtained.

  The request data output unit 3012 outputs a data request command to the database 250 based on the input request list. The background mesh model management unit 3013 stores the background mesh model received from the database 250.

  In the present embodiment, although the case is mainly described where the back end server 270 determines both the generation method of the virtual viewpoint image and the generation of the virtual viewpoint image, the present invention is not limited thereto. That is, the device that has determined the generation method may output data according to the determination result. For example, the front end server 230 determines the generation method to be used for generating the virtual viewpoint image based on the information related to the plurality of cameras 112 and the information output from the device for specifying the viewpoints related to the generation of the virtual viewpoint image. May be Then, the front end server 230 outputs the image data based on the photographing by the camera 112 and the information indicating the determined generation method to at least one of the storage device such as the database 250 and the image generation device such as the back end server 270. May be In this case, for example, the back end server 270 generates a virtual viewpoint image based on the information indicating the generation method output by the front end server 230. By the front end server 230 determining the generation method, it is possible to reduce the processing load due to the database 250 and the back end server 270 processing data for image generation in a method different from the determined method. On the other hand, when the back end server 270 determines the generation method as in the present embodiment, the database 250 holds data compatible with the plurality of generation methods, and thus, a plurality of virtual viewpoint images corresponding to the plurality of generation methods. Can be generated.

  FIG. 7 is a block diagram showing a functional configuration of the virtual camera operation UI 330. As shown in FIG. FIG. 8 is a diagram for explaining a virtual camera 8001. A virtual camera 8001 illustrated in FIG. 8A is a virtual camera that can perform shooting at a viewpoint different from that of any installed camera 112. That is, the virtual viewpoint image generated in the image processing system 100 is a captured image by the virtual camera 8001. In FIG. 8A, each of the plurality of sensor systems 110 arranged on the circumference has a camera 112. For example, by generating a virtual viewpoint image, it is possible to generate an image as if taken by the virtual camera 8001 near the soccer goal. A virtual viewpoint image which is a captured image of the virtual camera 8001 is generated by performing image processing on the images of the plurality of installed cameras 112. The operator (user) can obtain a photographed image from a free viewpoint by operating the position of the virtual camera 8001 or the like.

  The virtual camera operation UI 330 has a virtual camera management unit 8130 and an operation UI unit 8120. These may be implemented on the same device, or may be implemented separately on the device serving as the server and the device serving as the client. For example, in a virtual camera operation UI 330 used by a broadcasting station, a virtual camera management unit 8130 and an operation UI unit 8120 may be mounted on a workstation in a relay vehicle. Also, for example, the virtual camera management unit 8130 may be mounted on a web server, and the operation UI unit 8120 may be mounted on the end user terminal 190 to realize the same function.

  The virtual camera operation unit 8101 receives and processes an operation of the operator on the virtual camera 8001, that is, an instruction from the user for specifying a viewpoint related to generation of a virtual viewpoint image. The operation content of the operator is, for example, change of position (movement), change of posture (rotation), and change of zoom magnification. The operator uses input devices such as a joystick, a jog dial, a touch panel, a keyboard, and a mouse to operate the virtual camera 8001. The correspondence between the input by each input device and the operation of the virtual camera 8001 is determined in advance. For example, the “W” key of the keyboard is associated with an operation of moving the virtual camera 8001 one meter forward. Also, the operator can operate the virtual camera 8001 by designating a trajectory. For example, the operator writes a circle on the touch pad to designate a locus that the virtual camera 8001 turns on the circumference centered on the goal post. The virtual camera 8001 moves around the goalpost along the designated trajectory. At this time, the posture may be changed automatically so that the virtual camera 8001 always faces the goal post. The virtual camera operation unit 8101 can be used to generate a live image and a replay image. When generating a replay image, an operation is performed to specify time in addition to the position and orientation of the camera. In the replay image, for example, it is also possible to stop the time and move the virtual camera 8001.

  The virtual camera parameter derivation unit 8102 derives virtual camera parameters representing the position, attitude, and the like of the virtual camera 8001. The virtual parameters may be derived by operation or may be derived by reference to a look-up table or the like. As virtual camera parameters, for example, a matrix representing external parameters and a matrix representing internal parameters are used. Here, the position and orientation of the virtual camera 8001 are included in the external parameters, and the zoom value is included in the internal parameters.

  The virtual camera restriction management unit 8103 acquires and manages information for specifying a restricted area in which designation of a viewpoint based on an instruction accepted by the virtual camera operation unit 8101 is restricted. This information is, for example, a constraint on the position and orientation of the virtual camera 8001, the zoom value, and the like. Unlike the camera 112, the virtual camera 8001 can freely move the viewpoint and shoot, but it can not always generate images from all viewpoints. For example, even if the virtual camera 8001 is directed to a direction in which an object not shown in any camera 112 is shown, the captured image can not be acquired. In addition, when the zoom magnification of the virtual camera 8001 is increased, the image quality is deteriorated due to the restriction of the resolution. Therefore, a zoom magnification in a range that maintains a certain standard image quality may be used as the virtual camera restriction. The virtual camera constraints may be derived in advance from the arrangement of the cameras 112, for example. Also, the transmission unit 6120 may reduce the amount of transmission data according to the load on the network. By this data amount reduction, parameters relating to the photographed image change, and the range in which the image can be generated and the range in which the image quality can be maintained change dynamically. The virtual camera restriction management unit 8103 may be configured to receive information indicating the method used to reduce the data amount of output data from the transmission unit 6120, and dynamically update the virtual camera restriction according to the information. As a result, even if the amount of data is reduced by the transmission unit 6120, it is possible to maintain the image quality of the virtual viewpoint image at a constant reference.

  In addition, restrictions on the virtual camera 8001 are not limited to the above. In the present embodiment, the restricted area (area not satisfying the virtual camera restriction) in which the designation of the viewpoint is restricted is an operation state of a device included in the image processing system 100 and a parameter related to image data for generating a virtual viewpoint image. It changes according to at least one of them. For example, the restricted area changes in accordance with a parameter controlled such that the data amount of image data transmitted in the image processing system 100 falls within a predetermined range. The parameters include at least one of a frame rate of image data, a resolution, a quantization step, and an imaging range. For example, when the resolution of image data is reduced to reduce the amount of transmission data, the range of zoom magnifications that can maintain a predetermined image quality changes. In such a case, the virtual camera operation management unit 8103 acquires information specifying the restricted area that changes in accordance with the parameter, whereby the virtual camera operation UI 330 allows the user to specify the viewpoint in the range according to the change in parameter. It can be controlled to be done. The contents of the parameters are not limited to the above. Further, in the present embodiment, the image data whose data amount is controlled is data generated based on the difference between a plurality of photographed images by the plurality of cameras 112, but the present invention is not limited to this. It may be itself.

  Further, for example, the restricted area changes in accordance with the operation state of the apparatus included in the image processing system 100. Here, the apparatus included in the image processing system 100 includes, for example, at least one of a camera 112 and a camera adapter 120 that performs image processing on a captured image by the camera 112 to generate image data. The operation state of the device includes, for example, at least one of a normal state, a failure state, a start preparation state, and a restart state of the device. For example, when any of the cameras 112 is in a failure state or in a restart state, it may be possible that the viewpoint can not be specified at the peripheral position of the camera. In such a case, the virtual camera operation control unit 8103 acquires the information for specifying the restricted area that changes in accordance with the operation state of the device, whereby the virtual camera operation UI 330 is in a range corresponding to the change in the operation state of the device. Control can be performed so that the user can specify the viewpoint. The apparatus related to the change of the restricted area and the operation state thereof are not limited to the above.

  The collision determination unit 8104 determines whether the virtual camera parameter derived by the virtual camera parameter derivation unit 8102 satisfies the virtual camera constraint. If the constraint is not satisfied, for example, the operation input by the operator is canceled, and the virtual camera 8001 is controlled not to move from the position satisfying the constraint, or the virtual camera 8001 is returned to the position satisfying the constraint.

  The feedback output unit 8105 feeds back the determination result of the collision determination unit 8104 to the operator. For example, when the virtual camera constraint is not satisfied by the operation of the operator, the operator is notified of that. For example, it is assumed that the operator operates to move the virtual camera 8001 upward, but the destination does not satisfy the virtual camera constraint. In that case, the operator is notified that the virtual camera 8001 can not be moved further upward. As a notification method, there are methods such as sound, message output, screen color change, and lock of the virtual camera operation unit 8101. Furthermore, the position of the virtual camera may be automatically returned to a position that satisfies the constraints, which has the effect of leading to the operator's ease of operation. When the feedback is performed by image display, the feedback output unit 8105 causes the display unit to display an image based on display control according to the restricted area based on the information acquired by the virtual camera restriction management unit 8103. For example, in response to the instruction received by the virtual camera operation unit 8101, the feedback output unit 8105 causes the display unit to display an image indicating that the viewpoint corresponding to the instruction is within the restricted area. As a result, the operator can recognize that the specified viewpoint is within the restricted area and there is a possibility that the desired virtual viewpoint image can not be generated, and the viewpoint is respecified to the position outside the restricted area (the position satisfying the constraint). be able to. That is, in the generation of the virtual viewpoint image, it is possible to specify a viewpoint within a range which changes according to the situation. Note that the content displayed on the display unit by the feedback output unit 8105 is not limited to this. For example, an image may be displayed in which a portion corresponding to a restricted area in an area (for example, inside a stadium) which is a target of designation of a viewpoint is filled with a predetermined color. In the present embodiment, the display unit is an external display connected to the virtual camera operation UI 330. However, the present invention is not limited to this, and the display unit may exist inside the virtual camera operation UI 330.

  The virtual camera path management unit 8106 manages the path (virtual camera path 8002) of the virtual camera 8001 according to the operation of the operator. The virtual camera path 8002 is a sequence of information representing the position and orientation of each frame of the virtual camera 8001. It demonstrates with reference to FIG.8 (b). For example, virtual camera parameters are used as information representing the position and orientation of the virtual camera 8001. For example, one second's worth of information in the setting of the frame rate of 60 frames per second is a series of 60 virtual camera parameters. The virtual camera path management unit 8106 transmits the virtual camera parameters determined by the collision determination unit 8104 to the back end server 270. The back end server 270 generates a virtual viewpoint image and a virtual viewpoint sound using the received virtual camera parameters. The virtual camera path management unit 8106 also has a function of adding and holding virtual camera parameters to the virtual camera path 8002. For example, when a virtual viewpoint image and virtual viewpoint sound for one hour are generated using the virtual camera operation UI 330, virtual camera parameters for one hour are stored as a virtual camera path 8002. By storing this virtual camera path, it is possible to generate virtual viewpoint images and virtual viewpoint sound again by referring to the image information and virtual camera path stored in the secondary storage 2460 of the database later. become. That is, other users can reuse the virtual camera path generated by the operator who performs advanced virtual camera operation and the image information stored in the secondary storage 2460. A plurality of scenes corresponding to a plurality of virtual camera paths can also be stored in the virtual camera management unit 8130 so as to be selectable. When storing a plurality of virtual camera paths in the virtual camera management unit 8130, the script of the scene corresponding to each virtual camera path, the elapsed time of the game, the specified time before and after the scene, and meta information such as player information are also included. It can be input and stored. The virtual camera operation UI 330 notifies the back end server 270 of these virtual camera paths as virtual camera parameters.

  The end user terminal 190 can select the virtual camera path from the scene name, the player, the game elapsed time, and the like by requesting the back end server 270 for selection information for selecting the virtual camera path. The back end server 270 notifies the end user terminal 190 of selectable virtual camera path candidates, and the end user operates the end user terminal 190 to select a desired virtual camera path from a plurality of candidates. The end user terminal 190 can interactively enjoy the image distribution service by requesting the back end server 270 to generate an image according to the selected virtual camera path.

  The authoring unit 8107 provides an editing function when the operator generates a replay image. The authoring unit 8107 extracts a part of the virtual camera path 8002 held by the virtual camera path management unit 8106 as an initial value of the virtual camera path 8002 for replay image according to the user operation. As described above, the virtual camera path management unit 8106 holds meta information such as a scene name, a player, an elapsed time, and a designated time before and after a scene in association with the virtual camera path 8002. For example, a virtual camera path 8002 in which the scene name is a goal scene and the designated time before and after the scene is 10 seconds in total is taken out. Also, the authoring unit 8107 sets the playback speed for the edited camera path. For example, the virtual camera path 8002 is set to slow playback while the ball flies to the goal. When changing to an image from a different viewpoint, that is, when changing the virtual camera path 8002, the user operates the virtual camera 8001 again using the virtual camera operation unit 8101.

  The virtual camera image / sound output unit 8108 outputs the virtual camera image / sound received from the back end server 270. The operator operates the virtual camera 8001 while confirming the output image and sound. Note that, depending on the content of feedback by the feedback output unit 8105, the virtual camera image / sound output unit 8108 causes the display unit to display an image based on display control according to the restricted area. For example, when the position of the viewpoint designated by the operator is included in the restricted area, the virtual camera image / voice output unit 8108 is a virtual viewpoint image in which the position which is near the designated position and outside the restricted area is the viewpoint May be displayed. This reduces the need for the operator to respecify the viewpoint outside the restricted area.

  FIG. 9 is a block diagram showing the functional configuration of the end user terminal 190. As shown in FIG.

  The application management unit 10001 converts user input information input from a basic software unit 10002 described later into a back end server command of the back end server 270 and outputs the back end server command to the basic software unit 10002. In addition, the application management unit 10001 outputs, to the basic software unit 10002, an image drawing instruction for drawing the image input from the basic software unit 10002 in a predetermined display area.

  The basic software unit 10002 is, for example, an OS (Operating System), and outputs user input information input from a user input unit 10004 described later to the application management unit 10001. Also, it outputs an image and voice input from a network communication unit 10003 to be described later to the application management unit 10001 and outputs a back end server command input from the application management unit 10001 to the network communication unit 10003. Furthermore, the image drawing instruction input from the application management unit 10001 is output to the image output unit 10005.

  The network communication unit 10003 converts the back end server command input from the basic software unit 10002 into a LAN communication signal that can be communicated on a LAN cable, and outputs the LAN communication signal to the back end server 270. Then, the data is passed to the basic software unit 10002 so that the image and audio data received from the back end server 270 can be processed.

  The user input unit 10004 acquires user input information based on keyboard input (physical keyboard or soft keyboard) or button input, or user input information input from a user input device via a USB cable, and outputs it to the basic software unit 10002 Do.

  An image output unit 10005 converts an image based on an image display instruction output from the basic software unit 10002 into an image signal, and outputs the image signal to an external display, an integral display, or the like.

  The audio output unit 10006 outputs audio data based on the audio output instruction output from the basic software unit 10002 to an external speaker or an integrated speaker. The terminal attribute management unit 10007 manages the display resolution of the terminal 190, the image encoding codec type, and the terminal type (such as a smartphone or a large display).

  The service attribute management unit 10008 manages information on the service type provided to the end user terminal 190. For example, the types of applications installed in the end user terminal 190 and available image distribution services are managed.

  The charge management unit 10009 manages the number of image distribution scenes that can be received, etc., according to the user's registered payment status to the image distribution service and the charge amount.

<System workflow>
Next, a workflow in the case where a plurality of cameras 112 and microphones 111 are installed in a facility such as a stadium or a concert hall to perform photographing will be described.

  FIG. 10 is a flowchart showing the entire workflow in the first embodiment. The process of the workflow described below is realized by the control of the controller 300 unless there is a specific description. That is, control of the workflow is realized by the controller 300 controlling another device (for example, the back end server 270, the database 250, etc.) in the image processing system 100.

  Before the start of processing of the workflow, an operator (user) who installs or operates the image processing system 100 collects necessary information (prior information) before setting up and prepares a plan. In addition, it is assumed that the operator has installed equipment in the target facility in advance.

  In S1100, the control station 310 of the controller 300 receives the setting based on the prior information from the user. Next, in step S1101, each device of the image processing system 100 executes a process for confirming the operation of the system in accordance with a command issued from the controller 300 based on an operation from the user.

  In step S1102, the virtual camera operation UI 330 outputs an image and sound before the start of shooting for a competition or the like. Thereby, the user can confirm the sound collected by the microphone 111 and the image captured by the camera 112 before the competition or the like.

  In step S1103, the control station 310 of the controller 300 causes each microphone 111 to perform sound collection and causes each camera 112 to perform shooting. The shooting in this step includes sound collection using the microphone 111, but the invention is not limited thereto, and may be only shooting of an image. Then, in the case of changing the setting made in step S1101 or in the case of ending the photographing, the process proceeds to step S1104. Next, in S1104, if the setting made in S1101 is changed and shooting is continued, the process proceeds to S1105, and if shooting is completed, the process proceeds to S1106. The determination in S1104 is typically performed based on an input from the user to the controller 300. However, it is not limited to this example. In S1105, the controller 300 changes the setting made in S1101. The content of the change is typically determined based on the user input acquired in S1104. If it is necessary to stop the shooting in the setting change in this step, the shooting is once stopped, and the shooting is restarted after changing the setting. In addition, when it is not necessary to stop shooting, the setting is changed in parallel with shooting.

  In S1106, the controller 300 edits the images captured by the plurality of cameras 112 and the sound collected by the plurality of microphones 111. The editing is typically performed based on a user operation input via the virtual camera operation UI 330. The processes of S1106 and S1103 may be performed in parallel. For example, when a sports competition or a concert is distributed in real time (for example, the image of the competition is distributed during the competition), the shooting in S1103 and the editing in S1106 are performed simultaneously. In addition, when the highlight image in the sports competition is distributed after the competition, the editing is performed after the photographing is finished in S1104.

  FIG. 11 is a detailed flowchart of the installation process (S1101).

  In S1300, control station 310 receives user input regarding the presence or absence of excess or deficiency of installed equipment. The user can determine the presence or absence of the installed equipment by checking the presence or absence of the equipment information input in S1201 and the installed equipment. Next, in step S1301, the control station 310 executes installation confirmation processing of equipment determined to be insufficient in step S1300. That is, the user can install the lacking equipment between S1300 and S1301, and the control station 310 confirms that the insufficient equipment has been installed by the user.

  In step S1302, the control station 310 checks the pre-adjustment system operation whether the equipment installed in step S1301 is activated and operates normally. In the process of S1302, the user may check the system operation, and the user may input the check result to the control station 310.

  Here, if an error occurs in the excess or deficiency of equipment or operation, an error notification is sent to the control station 310 (S1303). The control station 310 is in a locked state which does not advance to the next step until the error is released. When the error state is released, the control station 310 is notified of normality (S1304), and the process proceeds to the next step. This makes it possible to detect an error at an early stage. After the confirmation, the process proceeds to step S1305 for the process related to the camera 112, and to step S1308 for the process related to the microphone 111.

  First, the camera 112 will be described. In S1305, the control station 310 performs adjustment of the installed camera 112. The adjustment of the camera 112 in this step refers to angle of view alignment and color alignment, and is performed for all the installed cameras 112. The adjustment in S1305 may be performed based on a user operation or may be realized by an automatic adjustment function.

  In the field angle alignment, adjustments of zoom, pan, tilt, and focus are performed in parallel, and the adjustment results are stored in the control station 310. In color matching, adjustments of IRIS, ISO / gain, white balance, sharpness, and shutter speed are simultaneously performed, and the adjustment results are stored in the control station 310.

  At S1306, the control station 310 adjusts all the installed cameras to be synchronized. The adjustment of synchronization in S1306 may be performed based on a user operation or may be realized by an automatic adjustment function. Furthermore, in step S1307, the control station 310 performs calibration at the time of camera installation. More specifically, the control station 310 adjusts so that the coordinates of all the installed cameras match the world coordinates. The details of the calibration process will be described later with reference to FIG. It should be noted that the communication check of the network path regarding synchronization with the control command of the camera 112 and the time server is also carried out. Then, it waits in the post-adjustment system operation normality confirmation process until the microphone adjustment proceeds (S1311).

  The process regarding the microphone 111 will be described. First, in S1308, the control station 310 performs adjustment of the installed microphone 111. The adjustment of the microphone 111 in this step refers to gain adjustment, and is performed for all the installed microphones. The adjustment of the microphone 111 in S1308 may be performed based on a user operation or may be realized by an automatic adjustment function.

  At S1309, the control station 310 adjusts all the installed microphones to be synchronized. Specifically, confirmation of the synchronization clock is performed. The adjustment of synchronization in S1309 may be performed based on a user operation or may be realized by an automatic adjustment function.

  In S1310, the control station 310 adjusts the position of the microphone 111 installed in the field among the installed microphones 111. The adjustment of the position of the microphone 111 in S1310 may be performed based on a user operation or may be realized by an automatic adjustment function. Note that the communication check of the network path regarding synchronization with the control command of the microphone 111 and the time server is also implemented.

  In step S1311, the control station 310 performs the post-adjustment system operation check for the purpose of confirming whether the cameras 112a to 112z and the microphones 111a to 111z are correctly adjusted. The process of S1311 may be performed based on a user instruction. If the camera 112 and the microphone 111 both confirm the system operation after adjustment, in step S1313, a normal notification is issued to the control station 310. On the other hand, when an error occurs, an error notification is sent to the control station 310 together with the type and individual number of the camera 112 or the microphone 111 (S1312). The control station 310 issues a readjustment instruction based on the type of the device in which the error has occurred and the individual number.

  Next, an operation example of the installation process (S1101) and the pre-shooting process (S1102) will be described. The image processing system 100 can switch and control the state of calibration at the time of installation and the state of normal imaging by changing the operation mode. In some cases, calibration of a specific camera is required during shooting, and in this case, two types of operations, shooting and calibration, are compatible.

  FIG. 12 is a sequence diagram of installation calibration processing. Although a description of notification of completion of reception of data and completion of processing in response to an instruction performed between devices is omitted here, it is assumed that some response is returned in response to the instruction.

  First, when the installation of the camera 112 is completed, the user instructs the control station 310 to perform calibration at installation. Then, the control station 310 instructs the front end server 230 and the camera adapter 120 to start calibration (S4100).

  When receiving the calibration start instruction, the front end server 230 determines that the image data received thereafter is data for calibration, and changes the control mode so that the calibration unit 2140 can process it (S4102a). In addition, when receiving the calibration start instruction, the camera adapter 120 does not perform image processing such as foreground / background separation, and shifts to a control mode for handling an uncompressed frame image (S4102b). Furthermore, the camera adapter 120 instructs the camera 112 to change the camera mode (S4101). The camera 112 having received this sets, for example, the frame rate to 1 fps. Alternatively, the camera 112 may be set to a mode for transmitting a still image instead of a moving image (S4102c). In addition, the frame rate may be controlled by the camera adapter 120 and a mode may be set in which a calibration image is transmitted.

  The control station 310 instructs the camera adapter 120 to acquire the zoom value and the focus value of the camera (S4103), and the camera adapter 120 transmits the zoom value and the focus value of the camera 112 to the control station 310 (S4103) S4104).

  Although only one camera adapter 120 and one camera 112 are described in FIG. 12, control relating to the camera adapter 120 and the camera 112 is performed for all the camera adapters 120 and all the cameras 112 in the image processing system 100. Be done. Therefore, S4103 and S4104 are executed by the number of cameras, and when the processes of S4103 and S4104 for all the cameras 112 are completed, the control station 310 can receive zoom values and focus values for all the cameras.

  The control station 310 transmits the zoom values and focus values for all the cameras received in S4104 to the front end server 230 (S4105). Next, the control station 310 notifies the front end server 230 of a shooting pattern for shooting during calibration (S4106).

  Here, in the shooting pattern, an attribute of a pattern name (for example, patterns 1 to 10) for distinguishing an image captured at another timing when a marker or the like serving as an image feature point is moved within the ground and captured a plurality of times Is added. That is, the front end server 230 determines that the calibration image data received after S4106 is a captured image in the captured pattern received in S4106. Then, the control station 310 instructs the camera adapter 120 to perform synchronous still image shooting (S4107), and the camera adapter 120 instructs the camera 112 to perform still image shooting synchronized with all the cameras (S4108). Then, the camera 112 transmits the photographed image to the camera adapter 120 (S4109).

  If there are a plurality of gaze point groups, the calibration image shooting of S4106 to S4111 may be performed for each gaze point group.

  Then, the control station 310 instructs the camera adapter 120 to transmit the image instructed to be shot in S4107 to the front end server 230 (S4110). Furthermore, the camera adapter 120 transmits the image received in S4109 to the front end server 230 specified as the transmission destination (S4111).

  For the calibration image transmitted in step S4111, image processing such as foreground / background separation is not performed, and the captured image is transmitted as it is without compression. Therefore, when all the cameras shoot at high resolution, or when the number of cameras increases, it may occur that all uncompressed images can not be transmitted simultaneously due to the limitation of the transmission band. As a result, the time required for calibration in the workflow may be long. In that case, in the image transmission instruction of S4110, the transmission instruction of the non-compressed image according to the pattern attribute of the calibration is sequentially issued to each camera adapter 120 one by one. Furthermore, in such a case, it is necessary to capture more feature points in accordance with the pattern attribute of the marker, so that calibration image capture is performed using a plurality of markers. In this case, image capture and non-compression image transmission may be performed asynchronously from the viewpoint of load distribution. In addition, the non-compressed image acquired by the image capturing for calibration is sequentially accumulated in the camera adapter 120 for each pattern attribute. Furthermore, by transmitting the non-compressed image in parallel according to the image transmission instruction in S4110, it is possible to reduce the workflow processing time and human error.

  When the process of S4111 is completed in all the cameras 112, the front end server 230 is in a state in which the photographed images for all the cameras can be received.

  As described above, when there are a plurality of shooting patterns, the processing of S4106 to S4111 is repeated for the number of patterns.

  Next, when all calibration imaging has been completed, the control station 310 instructs the front end server 230 to perform camera parameter estimation processing (S4112).

  When receiving the camera parameter estimation processing instruction, the front end server 230 performs camera parameter estimation processing using the zoom values and focus values for all cameras received in S4105 and the captured images for all cameras received in S4111. (S4113). Details of the camera parameter estimation process in S4113 will be described later. When there are a plurality of fixation points, the camera parameter estimation process of S4113 is performed for each fixation point group.

  Then, the front end server 230 transmits the camera parameters for all the cameras derived as a result of the camera parameter estimation process of S4113 to the database 250 for storage (S4114).

  The front end server 230 similarly transmits camera parameters for all cameras to the control station 310 (S4115). The control station 310 transmits camera parameters corresponding to each camera 112 to the camera adapter 120 (S4116), and the camera adapter 120 stores the received camera parameters of the own camera 112 (S4117).

  Then, the control station 310 confirms the calibration result (S4118). As a confirmation method, the numerical value of the derived camera parameter may be confirmed, or the calculation process of the camera parameter estimation process of S4114 may be confirmed, or an image is generated using the camera parameter. The image may be confirmed. Then, the control station 310 instructs the front end server 230 to finish the calibration (S4119).

In response to the calibration end instruction, the front end server 230 changes the control mode so as to determine that the image data received thereafter is not calibration data, contrary to the calibration start processing executed in S4101. (S4120)
By the above-described processing, as installation calibration processing, camera parameters for all cameras can be derived, and the derived camera parameters can be stored in the camera adapter 120 and the database 250.

  Also, the above-mentioned installation calibration process is performed after camera installation and before shooting, and there is no need to process it again if the camera is not moved, but when moving the camera (for example, note in the first half and the second half of the game) The same process is performed again to change the viewpoint, etc.).

  In addition, when the camera 112 moves above a predetermined threshold due to an accident such as ball collision during shooting, the camera 112 is transitioned from the shooting state to the calibration start state, and the above-described calibration is performed at the time of installation. Also good. In that case, it is necessary to keep the entire system in the calibration mode by maintaining the normal shooting state as the system and notifying the front end server 230 that only the camera 112 is transmitting the calibration image. It is possible to achieve the continuity of shooting. Furthermore, in the daisy chain transmission of the present system, when the uncompressed image for calibration is sent to the transmission band of the image data in normal imaging, the transmission band limit may be exceeded. In this case, the transmission priority of the non-compressed image is lowered or the non-compressed image is divided and transmitted. Furthermore, when the connection between the camera adapters 120 is 10 GbE or the like, by using the full-duplex feature, it is possible to secure a band by transmitting an uncompressed image in the opposite direction to the image data transmission of normal shooting. It has the effect of

  Further, when it is desired to change one of the plurality of fixation points, only the camera 112 of one fixation point group may perform the above-mentioned calibration process at the time of installation again. In that case, normal imaging and virtual viewpoint image generation can not be performed for the camera 112 of the target gaze point group during the calibration process. Therefore, the control station 310 is notified that the calibration process is in progress, and the control station 310 requests the virtual camera operation UI 330 to perform processing such as restriction of the viewpoint operation. The front end server 230 performs camera parameter estimation processing by performing control so as not to affect the processing of virtual viewpoint image generation.

  FIG. 13 is a detailed flowchart of the camera parameter estimation process. The calibration unit 2140 executes camera parameter estimation processing based on an instruction from the control station 310. At the time of starting this sequence, it is assumed that the calibration unit 2140 already holds the internal parameter map, stadium data, zoom values and focus values for all cameras, and calibration photographed images for all cameras. .

  First, the calibration unit 2140 identifies the camera 112 (S4201), identifies the corresponding zoom value and focus value, and derives an internal parameter initial value using the internal parameter map from the identified zoom value and focus value ( S4202). The processes of S4201 and S4202 are repeated until the derivation of the internal parameter initial value in S4202 is completed for all the cameras (S4203).

  Next, the calibration unit 2140 identifies the camera 112 again, identifies the corresponding calibration pickup image (S4204), and detects a feature point (image feature point) in the image (S4205). As image feature points, for example, markers prepared for calibration, pitch lines drawn in advance on the ground of the stadium, and edge portions of objects placed in advance (for example, soccer goals, player benches, etc.) Etc.

  The processes in S4204 and S4205 are repeated until the detection of the image feature points in S4205 for all the cameras is completed (S4206).

  Next, the calibration unit 2140 performs matching of the image feature points of the captured image for calibration in each camera 112 detected in S4205 (S4207). Then, it is determined whether the use feature score matched between the cameras 112 is equal to or less than a threshold (S4208). The threshold value of the feature number to be used in S4208 may be set in advance, or may be automatically derived according to the shooting conditions such as the number of cameras and the angle of view, and is at least necessary for external parameter estimation. The value is used.

  If it is determined in S4208 that the used feature score is not equal to or less than the threshold, the calibration unit 2140 performs external parameter estimation processing for each camera 112 (S4209). Then, as a result of the external parameter estimation processing in S4209, it is determined whether the reprojection error is equal to or less than the threshold (S4210). The threshold of the reprojection error used in S4210 may be set in advance, or may be automatically derived according to the shooting conditions such as the number of cameras, and a value corresponding to the accuracy of the virtual viewpoint image to be generated is used. Be

  If it is determined in step S4210 that the reprojection error is not equal to or less than the threshold value, the calibration unit 2140 determines that the error is large, and performs processing for deleting the erroneous detection of the image feature point in step S4205 and the incorrect matching of the image feature point in step S4207 S4211).

  For example, the calibration unit 2140 may automatically delete feature points having a large reprojection error as a method of determining the false detection and the false matching in S4211, or the user may manually perform a manual operation while viewing the reprojection error and the image. You may delete it by.

  Then, the calibration unit 2140 corrects the internal parameter with respect to the initial value of the internal parameter derived in S4202 (S4212). Then, the processing from S4208 to S4212 is repeated until the reprojection error becomes equal to or less than the threshold in S4210 within the range where the number of used feature points does not become equal to or less than the threshold in S4208.

  If it is determined in S4208 that the used feature score is equal to or less than the threshold value, the calibration unit 2140 determines that calibration has failed (S4213). In the case of calibration failure, measures such as starting again from calibration imaging are performed. The determination result of success or failure is notified to the control station 310 one by one, and the control station 310 centrally manages correspondence such as performing calibration processing after the failure time point.

  If it is determined in S4210 that the reprojection error is equal to or less than the threshold value, the calibration unit 2140 performs rigid body conversion from the camera coordinate system to the world coordinate system for the external parameter coordinates estimated in S4209 using stadium data S4214).

  As stadium data used here, the origin of each of X / Y / Z axes (for example, the center point of the center circle on the pitch) and coordinate values of a plurality of feature points in the stadium (for example, the intersection of pitch lines) , Coordinate values for performing rigid body transformation are defined.

  However, if there is no stadium data, or if the accuracy of the data is low, etc., world coordinates may be manually input to perform rigid body conversion, and the data for indicating world coordinates may be a calibration unit. It may be provided separately to 2140.

  In addition, since the world coordinates in the captured image for calibration are derived by performing the processing of S4214, the coordinates of the feature points in the stadium that are recorded in the stadium data in advance are more accurate using the derivation result. It may be updated to become

  By the above processing, camera parameters for all cameras can be derived as the camera parameter estimation processing flow, and the derived camera parameters can be stored in the camera adapter 120 and the database 250.

  Note that, in a system that generates virtual viewpoint images using captured images of a plurality of cameras, calibration processing (calibration at the time of installation) is necessary to estimate the position and orientation of each camera 112 when the camera 112 is installed.

  In the installation calibration, a process of obtaining camera parameters of each camera is performed. The camera parameters are composed of camera-specific internal parameters (focal length, image center, lens distortion parameter, etc.) and external parameters (rotation matrix, position vector, etc.) representing the position and orientation of the camera. When the installation calibration process is completed, the camera parameters of each camera are derived.

  Among the camera parameters, the internal parameter is a parameter that changes according to the zoom value and the focus value when the camera 112 and the lens are determined. Therefore, in the present system, before installing the camera 112 in the stadium, the internal parameters are derived by performing photographing necessary for deriving the internal parameters using the camera 112 and the lens. Then, when the zoom value and the focus value are determined when the camera 112 is installed in the stadium, internal parameters can be automatically derived. This is referred to herein as mapping internal parameters, and the result of the mapping is referred to as an internal parameter map.

  As the format of the internal parameter map, a plurality of patterns of internal parameters corresponding to the zoom value and the focus value may be recorded, or a formula of an operational expression capable of calculating the internal parameter value may be used. That is, the internal parameter map may be any one that can uniquely obtain the internal parameter according to the zoom value and the focus value.

  Further, parameter values obtained by the internal parameter map are used as initial values of the internal parameters. The internal parameter as the result of the camera parameter estimation process is a value corrected in the process of the camera parameter estimation process using the image captured for calibration after the camera 112 is installed in the stadium.

  Further, in the present embodiment, both the camera 112 and the lens to be installed are of the same model, and if the zoom value and the focus value are the same, the internal parameters are also the same. However, the present invention is not limited to this, and when there are individual differences in internal parameters even when the zoom value and the focus value are used, such as when using cameras 112 and lenses of multiple models, an internal parameter map for each model and camera 112 is used. You may hold it.

  By the way, there is a possibility that the data transmission amount may exceed the network transmission band limitation when transmitting the image frame of each camera 112 as the resolution of the camera image is increased. A method of reducing this fear will be described.

  FIG. 14 is a sequence diagram of generation processing of three-dimensional model information. Here, processing in which a plurality of camera adapters 120 (120a, 120b, 120c, and 120d) interlock to generate three-dimensional model information will be described. The order of processing is not limited to that shown in the figure.

  Although the image processing system 100 according to the present embodiment includes 26 cameras 112 and camera adapters 120, here, two cameras 112b and 112c, and four camera adapters 120a, 120b, 120c, and The description will be made focusing on 120d. The camera 112 b and the camera adapter 120 b, and the camera 112 c and the camera adapter 120 c are often connected. The camera 112 connected to the camera adapter 120a and the camera adapter 120d, the microphone 111 connected to each camera adapter 120, the camera platform 113, and the external sensor 114 are omitted. Further, it is assumed that the camera adapters 120a to 120d have completed time synchronization with the time server 290 and are in a shooting state.

  The camera 112b and the camera 112c each transmit the photographed image (1) and the photographed image (2) to the camera adapters 120b and 120c (F6301 and F6302). The camera adapters 120b and 120c perform calibration processing in the calibration control unit 6133 on the received captured image (1) or the received captured image (2) (F6303, F6304). The calibration process is, for example, color correction or shake correction. Although the calibration process is performed in the present embodiment, it may not be necessarily performed.

  Next, foreground / background separation processing is performed by the foreground / background separation unit 6131 on the captured image (1) or the captured image (2) that has undergone the calibration process (F6305, F6306).

  Next, compression is performed in the data compression / decompression unit 6121 on each of the separated foreground image and background image (F6307, F6308). The compression rate may be changed according to the degree of importance of the separated foreground image and background image. Also, in some cases, compression may not be performed. For example, the camera adapter 120 compresses at least the background image of the foreground image and the background image and outputs the compressed image to the next camera adapter 120 so that the compression ratio of the foreground image is lower than that of the background image. When the foreground image and the background image are both compressed, the foreground image including the important imaging target is subjected to lossless compression, and the background image not including the imaging target is subjected to lossy compression. As a result, it is possible to efficiently reduce the amount of data to be transmitted to the next camera adapter 120c or camera adapter 120d after this. For example, when shooting a field of a stadium where soccer, rugby, baseball, etc. are held, most of the image is composed of a background image, and the area of the foreground image of the player etc. is small. It is clearly stated here that significant reductions can be made.

  Furthermore, the camera adapter 120b or the camera adapter 120c may change the frame rate of the image to be output to the next camera adapter 120c or 120d depending on the importance. For example, even if the foreground image including the important imaging target is output at a high frame rate and the background image not including the imaging target is output at a low frame rate so that the output frame rate of the background image is lower than the foreground image. Good. This can further reduce the amount of data to be transmitted to the next camera adapter 120c or camera adapter 120d. Also, the compression rate and the transmission frame rate may be changed for each camera adapter 120 according to the installation location of the camera 112, the imaging location, and / or the performance of the camera 112, and the like. In addition, since a three-dimensional structure such as a spectator seat of the stadium can be confirmed in advance using a drawing, the camera adapter 120 may transmit an image obtained by removing the portion of the spectator seat from the background image. As a result, at the time of rendering, which will be described later, the image rendering focused on players in the game can be performed by utilizing the stadium three-dimensional structure generated in advance, and the amount of data transmitted and stored throughout the system can be reduced. The effect of being able to

  Next, the camera adapter 120 transfers the compressed foreground image and background image to the adjacent camera adapter 120 (F6310, F6311, F6312). Although the foreground image and the background image are simultaneously transferred in the present embodiment, each may be separately transferred.

  Next, the camera adapter 120b creates three-dimensional model information using the foreground image received from the camera adapter 120a and the foreground image separated in the foreground / background separation processing F6305 (F6313). Similarly, the camera adapter 120c also creates three-dimensional model information (F6314).

  Next, the camera adapter 120b transfers the foreground image and the background image received from the camera adapter 120a to the camera adapter 120c (F6315). The camera adapter 120c similarly transfers the foreground image and the background image to the camera adapter 120d. Although the foreground image and the background image are simultaneously transferred in the present embodiment, each may be separately transferred.

  Furthermore, the camera adapter 120c transfers the foreground image and the background image, which are created by the camera adapter 120a and received from the camera adapter 120b, to the camera adapter 120d (F6317).

  Next, each of the camera adapters 120a to 120c transfers the created three-dimensional model information to the next camera adapters 120b to 120d (F6318, F6319, F6320).

  Further, the camera adapters 120b and 120c transfer the sequentially received three-dimensional model information to the next camera adapters 120c and 120d (F6321, F6322). Furthermore, the camera adapter 120c transfers the three-dimensional model information created by the camera adapter 120a and received from the camera adapter 120b to the camera adapter 120d (F6323).

  Finally, the foreground image, the background image, and the three-dimensional model information created by the camera adapters 120a to 120d are sequentially transmitted between the networked camera adapters 120 and transmitted to the front end server 230.

  In the sequence diagram, the description of calibration processing of the camera adapter 120a and the camera adapter 120d, foreground / background separation processing, compression processing, and three-dimensional model information creation processing is omitted. However, in practice, the camera adapter 120a and the camera adapter 120d also perform the same processing as the camera adapter 120b and the camera adapter 120c, and create a foreground image, a background image, and three-dimensional model information. Further, although the data transfer sequence between the four camera adapters 120 has been described here, the same processing is performed even if the number of camera adapters 120 increases.

  As described above, among the plurality of camera adapters 120, the camera adapters 120 other than the last camera adapter 120 in the predetermined order extract a predetermined area from the image captured by the corresponding camera 112. Then, image data based on the extraction result is output to the next camera adapter 120 in the above-described predetermined order. On the other hand, the last camera adapter 120 outputs image data based on the extraction result to the front end server 230 in the above-described predetermined order. That is, the plurality of camera adapters 120 are connected in a daisy chain, and image data based on the result of extracting each predetermined area from the photographed image by each camera adapter 120 is input to the front end server 230 by the predetermined camera adapter 120. . By using such a data transmission method, it is possible to suppress fluctuations in processing load on the front end server 230 and transmission load in the network when the number of sensor systems 110 in the image processing system 100 fluctuates. The image data output from the camera adapter 120 is generated using the image data based on the above extraction result and the image data based on the extraction result of the predetermined area by the previous camera adapter 120 in a predetermined order. It may be data. For example, when each camera adapter 120 outputs image data based on the difference between the extraction result by itself and the extraction result by the previous camera adapter 120, it is possible to reduce the amount of transmission data in the system. In the above order, the last camera adapter 120 acquires extracted image data from the other camera adapter 120 based on the image data of the predetermined area extracted by the other camera adapter 120 from the image taken by the other camera 112. Then, image data corresponding to the extraction result of the predetermined area extracted by itself and the extracted image data acquired from another camera adapter 120 is output to the image computing server 200 for generating a virtual viewpoint image. .

  In addition, the camera adapter 120 divides the image captured by the camera 112 into a foreground portion and a background portion, and changes the compression rate and the frame rate to be transmitted according to, for example, the respective degrees of importance. As a result, it is possible to reduce the amount of transmission compared to the case of transmitting all the data captured by the camera 112 to the front end server 230. Also, each camera adapter 120 sequentially creates three-dimensional model information necessary for three-dimensional model generation. By this, it is possible to reduce the processing load of the server as compared to the case where all data are collected in the front end server 230 and all three dimensional model generation processing is performed by the server, and three dimensional model generation in more real time Can be made possible.

  FIG. 15 is a flowchart showing an operation of the camera adapter when receiving a captured image from the camera. Specifically, the camera adapter 120 generates the foreground image and the background image from the received captured image and transfers the foreground image and the background image to the subsequent camera adapter 120.

  The camera adapter 120 acquires a photographed image from the camera 112 connected to the camera adapter 120 (S6501). Next, processing is performed to separate the acquired captured image into a foreground image and a background image (S6502). The foreground image in the present embodiment is an image that is determined based on the detection result of a predetermined object with respect to the captured image acquired from the camera 112. The predetermined object is, for example, a person. However, the object may be a specific person (such as a player, a director, and / or an umpire), or an object such as a ball or a goal, for which an image pattern is predetermined. Also, a moving body may be detected as an object.

  Next, compression processing of the separated foreground image and background image is performed (S6503). Lossless compression is performed on the foreground image, and the foreground image maintains high image quality. Lossy compression is performed on the background image to reduce the amount of transmission data.

  Next, the camera adapter 120 transfers the compressed foreground image and background image to the next camera adapter 120 (S6504). As for the background image, transfer frames may be thinned out and transferred instead of transferring each frame. For example, when the captured image is 60 fps, the foreground image is transmitted every frame, but the background image is transmitted only one frame out of 60 frames per second. As a result, there are special effects that can reduce the amount of transmission data.

  Also, the camera adapter 120 may add meta information when transferring the foreground image and the background image to the next camera adapter 120. For example, the identifier of the camera adapter 120 or the camera 112, the position (xy coordinates) of the foreground image in the frame, the data size, the frame number, the photographing time, and the like are given as meta information. Also, gaze point group information for identifying a gaze point, data type information for identifying a foreground image and a background image, and the like may be added. However, the contents of the data to be given are not limited to these, and other data may be given.

  In addition, when the camera adapter 120 transmits data through the daisy chain, the transmission processing load on the camera adapter 120 can be reduced by selectively processing only the photographed image of the camera 112 highly correlated with the camera 112 connected to the camera adapter 120. It can be reduced. Further, in the daisy chain transmission, robustness can be ensured by configuring the system so that data transmission between the camera adapters 120 does not stop even if a failure occurs in any of the camera adapters 120.

  FIG. 16 is a flowchart showing the operation of the camera adapter when data is received from an adjacent camera adapter.

  First, the camera adapter 120 receives data from the adjacent camera adapter 120 (S6601). The camera adapter 120 determines whether its own transfer mode is the bypass control mode (S6602). The details of the bypass control mode will be described later with reference to FIG.

  In the case of the bypass control mode, the camera adapter 120 transfers data to the next camera adapter 120 (S6611). If the bypass control mode is not set, the packet of the received data is analyzed (S6603).

  As a result of analyzing the packet, if the camera adapter 120 determines that the packet is a bypass transmission control target packet (Yes in S6604), it transfers data to the next camera adapter 120 (S6610). The packet subjected to bypass transmission control is, for example, image data not used for three-dimensional model information generation, a control message described later, or a message related to time correction. The details of the bypass transmission control will be described later with reference to FIG.

  If it is determined that the camera adapter 120 is not the bypass transmission control target, the camera adapter 120 determines the data type (S6605), and performs processing according to the data type.

  If the type of data is a control message packet addressed to the camera adapter 120 from the control station 310, the control message is analyzed and processing is performed based on the analysis result (S6606). The same applies when the source of the control message is not the control station 310 but another node. In addition, not only when the packet is addressed to its own camera adapter 120 but also when addressed to a fixation point group to which the camera adapter 120 belongs. Further, examples of processing performed by the camera adapter 120 include control of the microphone 111 connected to the camera adapter 120, control of the camera 112 and camera platform 113, and control of the camera adapter 120 itself. The camera adapter 120 returns the control result to the sender or designated node according to the content of the control message. If the packet is a control message addressed to a group, the control message is transferred to the next camera adapter 120.

  Next, when the data type relates to time correction, the camera adapter 120 performs time correction processing (S6607). For example, based on PTP processing with the time server 290, it performs its own time correction. Then, the word clock supplied to the microphone 111 and the camera 112 is corrected based on the corrected time. If the timing of the word clock is changed at one time when the correction amount of time is large, the voice and the image quality are affected. Therefore, the time may be gradually corrected based on the change amount set in advance. Also, the camera adapter 120 transfers the created three-dimensional model information and the foreground image used for creating the three-dimensional model information to the next camera adapter 120 for transmission to the front end server 230.

  Next, the camera adapter 120 performs three-dimensional model information creation processing when the data type is a foreground image or a background image (S6608).

  FIG. 17 is a diagram for explaining a gaze point group. Each camera 112 is installed such that the optical axis is directed to a particular gaze point 6302. The cameras 112 classified into the same gaze point group 6301 are installed to face the same gaze point 6302.

  FIG. 17 shows an example in which two fixation points 6302 of fixation point A (6302A) and fixation point B (6302B) are set and nine cameras (112a to 112i) are installed. Four cameras (112a, 112c, 112e and 112g) face the same fixation point A (6302A) and belong to fixation point group A (6301A). The remaining five cameras (112b, 112d, 112f, 112h and 112i) face the same fixation point B (6302B) and belong to the fixation point group B (6301B).

  Here, among the cameras 112 belonging to the same gaze point group 6301, a set of cameras 112 (the number of connection hops is small) is expressed as “logically adjacent”. For example, the camera 112 a and the camera 112 b are physically adjacent but not logically adjacent because they belong to different gaze point groups 6301. Logically adjacent to the camera 112a is a camera 112c. On the other hand, the camera 112 h and the camera 112 i are not only physically adjacent but also logically adjacent.

  Depending on whether or not the physically adjacent cameras 112 are logically adjacent, different processes are performed in the camera adapter 120. Specific processing will be described below.

  FIG. 18 is a diagram for explaining bypass transmission control. The bypass transmission control is a function of bypassing transmission data according to a gaze point group to which each camera adapter 120 belongs. Descriptions of functional units constituting the external device control unit 6140, each image processing unit 6130, the transmission unit 6120, and the network adapter 6110 are omitted.

  In the image processing system 100, the number of camera adapters 120 and the setting of which camera adapter 120 belongs to which gaze point group can be changed. In FIG. 18, it is assumed that the camera adapter 120 g, the camera adapter 120 h, and the camera adapter 120 n belong to the fixation point group A, and the camera adapter 120 i belongs to the fixation point group B.

  A route 6450 indicates a transmission route of the foreground image generated by the camera adapter 120g, and the foreground image is finally transmitted to the front end server 230. In the drawing, the description of the background image, the three-dimensional model information, the control message, and the foreground image created by the camera adapter 120h, the camera adapter 120i and the camera adapter 120n is omitted.

  The camera adapter 120h receives the foreground image created by the camera adapter 120g via the network adapter 6110h, and determines the routing destination by the transmission unit 6120h. If it is determined that the camera adapter 120g from which the received foreground image is created belongs to the same gaze point group (here, group A), the transmission unit 6120h transfers the received foreground image to the image processing unit 6130h. When three-dimensional model information is generated in the image processing unit 6130 h based on the foreground image generated and transmitted by the camera adapter 120 g, the foreground image of the camera adapter 120 g is transferred to the next camera adapter 120 i.

  Next, the camera adapter 120i receives the foreground image created by the camera adapter 120g from the camera adapter 120h. If the transmission unit 6120i of the camera adapter 120i determines that the fixation point group to which the camera adapter 120g belongs differs, the transmission unit 6120i does not transfer it to the image processing unit 6130i, but transfers it to the next camera adapter 120.

  Next, the camera adapter 120 n receives the foreground image created by the camera adapter 120 g via the network adapter 6110 n, and determines the routing destination by the transmission unit 6120 n. The transmission unit 6120 n determines that the camera adapter 120 n is the same gaze point group as the camera adapter 120 g. However, when it is determined by the image processing unit 6130 n that the foreground image of the camera adapter 120 g is not a foreground image necessary for generating three-dimensional model information, the foreground image is directly transferred to the next camera adapter 120 via the daisy chain network. Be done.

  As described above, the transmission unit 6120 of each camera adapter 120 determines whether the received data is data necessary for creating three-dimensional model information by image processing in the image processing unit 6130. If it is determined that the data is not data necessary for image processing, that is, data having a low correlation with its own camera adapter 120, the data is transmitted to the next camera adapter 120 without being transferred to the image processing unit 6130. That is, in the data transmission via the daisy chain 170, necessary data is selected in each camera adapter 120, and processing of sequentially generating three-dimensional model information is performed. As a result, processing load and processing time related to data transfer from data reception to transfer in the camera adapter 120 can be shortened.

  FIG. 19 is a diagram for explaining bypass control of the camera adapter 120b. Note that descriptions of functional units constituting the external device control unit 6140, the respective image processing units 6130, the transmission unit 6120, and the network adapter 6110 are omitted.

  The bypass control is a function of transferring data received from the camera adapter 120c to the next camera adapter 120a without performing routing control by the data routing processing unit 6122 of the transmission unit 6120.

  For example, the camera adapter 120 b causes the network adapter 6110 to start bypass control when the state of the camera 112 b is in the state of shooting stop, calibration or error processing. Further, for example, even when an operation failure or the like of the transmission unit 6120 or the image processing unit 6130 occurs, the bypass control is activated. In addition, the network adapter 6110 may detect the state of the transmission unit 6120 and actively transition to the bypass control mode. Note that a sub CPU that detects that the transmission unit 6120 or the image processing unit 6130 is in an error state or a stop state is deployed in the camera adapter 120b, and the network adapter 6110 is subjected to bypass control when an error is detected by the sub CPU. Processing may be added. This has the effect of being able to independently control the fault state and bypass control of each functional block.

  Also, even if the camera adapter 120 transitions from the bypass control mode to the normal communication mode when the state of the camera 112 transitions from the calibration state to the imaging state, or when the transmission unit 6120 or the like recovers from a malfunction. Good.

  With this bypass control function, the camera adapter 120 can perform data transfer at high speed, and can transfer data to the next camera adapter 120 a even if an unexpected failure or the like occurs and it is not possible to make a determination related to data routing. .

  In the present system, foreground images, background images, and three-dimensional model information are transmitted between a plurality of camera adapters 120 connected in a daisy chain and input to the front end server 230. Here, when an event in which the foreground area becomes extremely large in the photographed image, for example, an opening ceremony or the like in which all players gather is photographed, the amount of data of the foreground image to be transmitted is more than when photographing a regular competition. It will be huge. Therefore, a method for controlling so that the amount of data transmitted by the daisy chain does not exceed the transmission band will be shown below.

  The flow of processing in which the transmission unit 6120 outputs data in the camera adapter 120 will be described using FIG. 20 and FIG.

  FIG. 20 exemplarily shows the flow of data between camera adapters 120a, 120b and 120c. The camera adapter 120a and the camera adapter 120b, and the camera adapter 120b and the camera adapter 120c are often connected. Further, a camera 112 b is connected to the camera adapter 120 b, and the camera adapter 120 c is connected to the front end server 230. A data output processing flow of the transmission unit 6120 of the camera adapter 120b will be described.

  Photographed data 6720 is input from the camera 112b to the transmission unit 6120 of the camera adapter 120b, and input data 6721 and 6722 subjected to image processing are input from the camera adapter 120a. The transmission unit 6120 performs processing such as output to the image processing unit 6130, compression, setting of a frame rate, and packetization on the input data, and outputs the data to the network adapter 6110. .

  FIG. 21 is a flowchart of output processing in the transmission unit 6120. The transmission unit 6120 executes the step (S6701) of acquiring the data amount of the image processing result for each of the input data 6721 and 6720 from the image processing unit 6130.

  Next, the step (S6702) of acquiring the data amount of the input data 6722 from the camera adapter 120a is executed. Next, the step (S6703) of deriving the amount of output data to the camera adapter 120c is performed according to the data type of the input data.

  Next, the transmission unit 6120 compares the amount of output data with a predetermined transmission band restriction amount to confirm the transmission possibility. Specifically, it is determined whether the amount of data to be output to the network adapter 6110 exceeds a threshold of the amount of output data specified in advance (S 6704). The threshold may be provided for each data type (here, a foreground image, a background image, whole scene frame data, three-dimensional model information, and the like can be mentioned). The amount of data to be output is derived based on the compression result when the transmission unit 6120 compresses the data. The threshold value of the amount of output data is preferably set in consideration of overhead such as header information and error correction information at the time of packetization.

  If the transmission unit 6120 determines that the amount of output data does not exceed the threshold value, it performs normal transfer to output input data to the network adapter 6110 (S6712). If it is determined that the output data amount has exceeded the threshold (Yes in S6704), if the data input to the transmission unit 6120 is image data, a policy for when the output data amount is over is acquired (S6705). Then, based on the acquired policy, at least one of a plurality of processes (S6707 to S6711) described below is selected and executed (S6706). The transmission unit 6120 may normally transfer data other than image data related to time correction and data related to a control message. Also, the message may be dropped according to the type and priority of the message. Overflow of data transfer can be suppressed by reducing the amount of output data.

  As one of the processes executed by the transmission unit 6120, the frame rate of the image data is dropped and output to the network adapter 6110 (S6707). By decimating and transmitting frames, the amount of data is reduced. However, when following fast-moving objects, there is a possibility that the image quality may be inferior as compared with the case of outputting at a high frame rate, so the applicability of this method is determined according to the target shooting scene.

  As another process, the transmission unit 6120 reduces the resolution of the image data and outputs it to the network adapter 6110 (S6708). Since this process affects the image quality of the output image, a policy is set according to the type of end user terminal. For example, when outputting to a smart phone, the resolution is greatly reduced to reduce the data amount, and when outputting to a high resolution display or the like, policy setting regarding adaptive resolution conversion such as decreasing the resolution is set.

  As another process, the transmission unit 6120 increases the compression rate of the image data and outputs the image data to the network adapter 6110 (S6709). Here, with respect to the input image data, data amount reduction is achieved according to the restoration performance request such as lossless compression or lossy compression, that is, the image quality request.

  As another process, the transmission unit 6120 stops the output of the shooting data 6720 from the image processing unit 6130 (S6710). Here, the output of the image data subjected to the image processing is stopped to reduce the data amount. When a sufficient number of cameras 112 are deployed, there may be cases where not all the cameras 112 of the same gaze point group are required in the generation of virtual viewpoint images. For example, the present control is applied when it is known in advance that no blind spot will occur even if the camera 112 is reduced in photographing the entire field of the stadium. That is, it is possible to secure a transmission band by selecting a camera that does not transmit image data on condition that failure of the image in a post process does not occur.

  Further, as another process, the transmission unit 6120 stops the output of the input data 6721 from the image processing unit 6130 or stops the output of only an image of a part of the camera adapters 120 (S6711). In addition to the above, when three-dimensional model information can be generated using an input image from another camera adapter 120, the output of the foreground image or the background image from the other camera adapter 120 is stopped, and the three-dimensional image is generated. The data volume may be reduced by controlling only the model information.

  The method used to reduce the amount of output data is notified to the back end server 270, the virtual camera operation UI 330, and the control station 310 via the front end server 230 at a later stage (S6713). In the present embodiment, the flow is branched so that any of control processing such as frame rate, resolution, compression rate, and data stop is performed according to the policy, but the present invention is not limited to this. It should be clearly stated that the effect of reducing the amount of data can be obtained by executing a combination of two or more of these controls. In addition, notification of the control process is performed in S6713. By this notification, in the virtual camera operation UI 330, for example, when a sufficient resolution can not be obtained in terms of image quality as a result of increasing the compression rate, it is possible to place a restriction on the zoom operation. Furthermore, it is shown here that even after the transmission band constraint amount over processing, the excess of the output data amount is checked successively, and if the data amount is stabilized, the transmission processing policy can be returned to the original set value.

  Thus, there is an effect that transmission satisfying the transmission band constraint can be realized by performing the transmission control processing according to the state with respect to the problem of exceeding the transmission band of the daisy chain.

  FIG. 23 is a flowchart of various processes in the image processing unit 6130 of the camera adapter 120. Prior to the processing of FIG. 23A, the calibration control unit 6133 first performs color correction processing on an input image to suppress variation in color among cameras, and against blurring due to camera vibration. Shake correction processing for stabilizing the position of the image. The shake correction process is, for example, an electronic image stabilization process.

  In the color correction processing, processing such as adding an offset value to the pixel value of the input image is performed based on the parameters received from the front end server 230. In the shake correction process, the shake amount of the image is estimated based on output data from a sensor such as an acceleration sensor or a gyro sensor built in the camera. Then, shift of the image position with respect to the input image and rotation processing of the image are performed based on the estimated blur amount, thereby suppressing blur between frame images. Note that other methods may be used as the blur correction method. For example, a method based on image processing that estimates and corrects a moving amount of an image by comparing a plurality of temporally consecutive frame images, a method realized inside a camera such as a lens shift method and a sensor shift method, etc. May be.

  The background updating unit 5003 performs processing of updating the background image 5002 using the input image and the background image stored in the memory.

  FIG. 22 is a diagram for explaining a background image. FIG. 22A shows an example of the background image. The update process is performed on each pixel. The processing flow is shown in FIG.

  First, at S5001, the background updating unit 5003 derives, with respect to each pixel of the input image, the difference between the pixel and the pixel at the corresponding position in the background image. Next, in S5002, it is determined whether the difference is smaller than a predetermined threshold K. If the difference is smaller than K, it is determined that the pixel is a background (YES in S5002). In step S5003, the background updating unit 5003 derives a value obtained by mixing the pixel value of the input image and the pixel value of the background image at a constant ratio. In step S5004, the pixel value in the background image is updated with the derived value.

  On the other hand, FIG. 22 (b) is a view showing an image in which a person appears in FIG. 22 (a) which is a background image. In such a case, focusing on the pixel at which the person is located, the difference in pixel value with respect to the background is large, and the difference becomes K or more in S5002. In that case, the change of the pixel value is large, so it is determined that an object other than the background is displayed, and the background image 5002 is not updated (NO in S5002). Various other methods can be considered for the background updating process.

  Next, the background cutout unit 5004 reads a part of the background image 5002 and transmits the part to the transmission unit 6120. When shooting a game such as soccer on a stadium or the like, when a plurality of cameras 112 are arranged so that the entire field can be shot without blind spots, most of the background information overlaps between the cameras 112. Since the background information is huge, it is possible to reduce the amount of transmission by deleting and transmitting the overlapping part in terms of transmission band constraints. The flow of the process is shown in FIG. In step S5010, the background cutout unit 5004 sets the central portion of the background image, as in the partial region 3401 surrounded by a dotted line illustrated in FIG. 22C, for example. That is, the partial area 3401 is a background area in which the own camera 112 takes charge of transmission, and the other background areas are in charge of transmission by the other cameras 112.

  In step S5011, the background cutout unit 5004 reads out the partial area 3401 of the set background image. Then, in step S5012, the data is output to the transmission unit 6120. The output background image is collected by the image computing server 200 and used as a texture of the background model. The position from which the background image 5002 is cut out in each camera adapter 120 is set according to a predetermined parameter value so that there is no shortage of texture information for the background model. Usually, in order to reduce the amount of transmission data, the area to be cut out is set to be the minimum necessary. This has the effect of being able to reduce the amount of transmission of a vast amount of background information, and a system capable of coping with high resolution can be achieved.

  Next, the foreground separation unit 5001 performs processing to detect a foreground area (an object such as a person). A flow of foreground area detection processing performed for each pixel is shown in FIG. For foreground detection, a method using background difference information is used. First, in step S5005, the foreground separation unit 5001 derives the difference between each pixel of the newly input image and the pixel at the corresponding position in the background image 5002. In step S5006, it is determined whether the difference is larger than the threshold L. Here, assuming that the newly input image is as shown in, for example, FIG. 22 (b) with respect to the background image 5002 shown in FIG. 22 (a), each pixel of the area where the person appears The difference becomes large in. If the difference is larger than the threshold L, the pixel is set as the foreground in S5007. In the foreground detection method using background difference information, various devices for detecting the foreground with higher accuracy are considered. In addition, there are various methods, such as methods using feature quantities and machine learning, for foreground detection.

  The foreground separation unit 5001 executes the processing described above with reference to FIG. 23B for each pixel of the input image, and then performs processing for determining and outputting the foreground region as a block. The flow of processing is shown in FIG. In S5008, the foreground area in which a plurality of pixels are connected is set as one foreground image for the image in which the foreground area is detected. For example, a region growing method is used as a process of detecting a region in which pixels are connected. Since the region growing method is a known algorithm, the detailed description is omitted. After the foreground regions are grouped as foreground images in S5008, the foreground images are sequentially read out and output to the transmission unit 6120 in S5009.

  Next, the three-dimensional model information generation unit 6132 generates three-dimensional model information using the foreground image. When the camera adapter receives the foreground image from the adjacent camera, the foreground image is input to the other camera foreground reception unit 5006 via the transmission unit 6120. The flow of processing executed by the three-dimensional model processing unit 5005 when a foreground image is input is shown in FIG. Here, when the image computing server 200 collects captured image data of the camera 112, starts image processing, and generates a virtual viewpoint image, there may be a large amount of calculation and a long time for image generation. In particular, the amount of calculation in three-dimensional model generation may be significantly increased. Thus, in FIG. 23E, in order to reduce the amount of processing in the image computing server 200, a method of sequentially generating three-dimensional model information while transmitting data by daisy-chaining the camera adapters 120 will be described.

  First, in step S5013, the three-dimensional model information generation unit 6132 receives a foreground image captured by another camera 112. Next, at 5014, the three-dimensional model information generation unit 6132 checks whether the camera 112 that has captured the received foreground image belongs to the same gaze point group as the subject camera 112 and is an adjacent camera. If S5014 is YES, the process proceeds to S5015. In the case of NO, it is determined that there is no correlation with the foreground image of the other camera 112, and the process ends without performing processing. Further, in S5014, it is confirmed whether the camera is an adjacent camera, but the method of determining the correlation between the cameras 112 is not limited to this. For example, the method is similar to the method in which the three-dimensional model information generation unit 6132 obtains and sets the camera number of the correlated camera 112 in advance, and acquires and processes the image data only when the image data of the camera 112 is transmitted. An effect is obtained.

  Next, in S5015, the three-dimensional model information generation unit 6132 derives depth information of the foreground image. Specifically, first, the foreground image received from the foreground separation unit 5001 is associated with the foreground image of another camera 112, and then each foreground is matched based on the coordinate value of each pixel and the camera parameter. The depth information of each pixel on the image is derived. Here, for example, a block matching method is used as a method of associating images. Since the block matching method is a well-known method, the detailed description is omitted. Further, there are various methods for improving the performance by combining feature point detection, feature amount calculation, matching processing and the like as a method of association, and any method may be used.

  Next, in S5016, the three-dimensional model information generation unit 6132 derives three-dimensional model information of the foreground image. Specifically, for each pixel of the foreground image, the world coordinate value of the pixel is derived based on the depth information derived in S5015 and the camera parameter stored in the camera parameter receiving unit 5007. Then, one set of point data of the three-dimensional model configured as a point group is set by setting the world coordinate value and the pixel value as a set. According to the above processing, part of point cloud information of the three-dimensional model obtained from the foreground image received from the foreground separation unit 5001 and part of point cloud of the three-dimensional model obtained from the foreground image of another camera 112 Information is obtained. Then, in step S5017, the three-dimensional model information generation unit 6132 adds the camera number and the frame number to the obtained three-dimensional model information as meta information (the meta information may be, for example, a time code or an absolute time). Output to

  Thus, even when the camera adapters 120 are connected by a daisy chain and a plurality of fixation points are set, three-dimensional model information can be sequentially generated. That is, while data is transmitted by the daisy chain, image processing can be performed according to the correlation between the cameras 112 to sequentially generate three-dimensional model information. As a result, the processing can be speeded up.

  In the present embodiment, each processing described above is executed by hardware such as FPGA or ASIC mounted on the camera adapter 120, but may be executed by software processing using, for example, a CPU, GPU, DSP, etc. Good. Although three-dimensional model information generation is executed in the camera adapter 120 in the present embodiment, the image computing server 200 in which all foreground images from each camera 112 are collected may generate three-dimensional model information.

  FIG. 24A is a flowchart of processing in the deviation detection notification unit. FIG. 24B is a detailed flowchart of foreground / background separation processing in step S202. The shift detection notification unit 6134 has a function of detecting a shift of an image from the background image separated by the foreground / background separation unit 6131.

  First, FIG. 24 (a) will be described. In step S201, an image is captured after calibration at the time of setting. Next, in step S202, the foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the temporal correlation and the spatial correlation with the image obtained in step S201. Details regarding this temporal correlation and foreground / background separation processing for separating foreground and background from spatial correlation will be described later. Then, the background image 5002 is stored in the reference background image storage unit 5008 as a reference image. In addition, for example, in the case where shooting in S201 is performed before the stadium opening (a player or the like does not exist on the field and there is no spectator on the stand), the image obtained in S201 is used as a reference image as it is You may

  Next, in step S203, the current image is captured at system startup or at regular timing. Next, in step S204, the foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the spatial correlation as described above in the foreground separation unit 5001, and compares the background image 5002 for comparison. Are stored in the comparison current background image storage unit 5009 as the current image of. Next, in step S205, the shift detection unit 5010 compares the reference image stored in the reference background image storage unit 5008 with the comparison current image stored in the comparison current background image storage unit 5009 to detect the shift. As a method of comparison, a method using vectors between corresponding feature points, or other known techniques can be used. If the deviation is larger than a predetermined reference value, it is determined that the installation state of the camera has changed, and an alarm is notified and alarm information is transmitted to the transmission unit 6120.

  Next, FIG. 24B will be described. The process of FIG. 24B is performed on each pixel of the input image. In step S <b> 20201, for the target pixel of the input image, the difference between the temporal correlation and the spatial correlation with the background image is calculated. Specifically, when calculating the temporal correlation difference, the difference between the value of the target pixel of the input image and the value of the corresponding pixel of the background image with the image taken in the past by the same camera as the input image as the background image Is calculated. On the other hand, when calculating the spatial correlation difference, the difference between the value of the target pixel of the input image and the value of the corresponding pixel of the background image with the image taken at substantially the same timing by the camera different from the input image as the background image. Is calculated. Next, in step S20202, a determination is made as to the temporal correlation difference, and if the difference is smaller than the predetermined value L, the process proceeds to the next step, otherwise the process ends. Next, in step S20203, the spatial correlation difference is determined, and if the difference is smaller than the predetermined value D, the process proceeds to the next step, otherwise the process ends. Next, in step S20204, the target pixel of the input image determined to have both the temporal correlation difference and the spatial correlation difference within the reference range is stored as a background pixel to be separated. By performing the process of FIG. 24B on all the pixels of the input image, it is possible to separate the background based on the spatial correlation and the temporal correlation from the input image.

  The foreground / background separation processing in step S204 in FIG. 24A is the same. Note that at least one of foreground / background separation in S202 and S204 may be foreground / background separation based on only temporal correlation or foreground / background separation based on only spatial correlation. Here, the difference detection in the temporal correlation may be, for example, a method of generating background data based on information of an image of a predetermined time in the past and detecting a difference from the current image. Further, the detection method is not limited to this method, and a method such as a method using feature amounts or machine learning may be used. On the other hand, foreground / background separation processing in spatial correlation referred to here may be a method of extracting a common object and a region from spatial correlation. As a specific method, there is a method of retaining only a background as a given 3D model having 3D information in advance and comparing it with the current image to extract only the matching background region. As a similar method, it may be a method of separating the foreground located at a short distance from the background located at a long distance by means of simultaneously measuring the distance corresponding to each object. Also, foreground / background separation processing in spatial correlation is not limited to this. By this processing, it is possible to separate only the background at a long distance from the image captured at the same time.

  FIG. 25 exemplarily shows foreground / background separation processing using temporal correlation and spatial correlation. Here, as shown in FIG. 25A, the camera 110 is photographing the direction of the field of the stadium and the audience seat. A schematic imaging range imaged by the camera 110 is indicated by K. FIGS. 25 (b1) to 25 (b4) are diagrams for explaining the image change when the foreground / background separation process is applied to the camera shot image in the shooting range. FIG. 25 (b1) shows an image to be subjected to foreground / background separation. The person included in this image is temporally changed in position, but other structures are temporally invariant. Therefore, if the difference between the image to be separated shown in FIG. 25 (b1) and the image photographed by the same camera at another time is calculated, and foreground / background separation processing by temporal correlation is performed, as shown in FIG. As in (b2), a moving person is separated as the foreground, and a structure is separated as the background.

  On the other hand, among the structures, there are a goal in front of the camera placed relatively close to the camera and a floor, beams, and columns of a spectator seat at a distance from the camera. When the photographed images of a plurality of cameras installed adjacent to each other are compared, the difference in the position of an object located at a short distance from the camera becomes large, but the difference in the position of an object located at a distance from the camera is relatively small. Therefore, by calculating the difference between a plurality of photographed images photographed by a plurality of cameras at substantially the same timing, it is possible to separate a distant audience seat from a goal or a person in front. The foreground / background separation method using this spatial correlation may be a method using a background as a 3D model having 3D information, a method using distance information, or the like, as described above. When foreground / background separation processing by spatial correlation is performed on the image in FIG. 25 (b1), the goal and the player in front are separated as foreground and the audience seat is separated as background as shown in FIG. 25 (b3). Ru.

  Furthermore, if foreground / background separation processing by spatial correlation is performed on the image of FIG. 25 (b2) separated from FIG. 25 (b1) by temporal correlation, the goal is separated as shown in FIG. 25 (b4). The background of the seat floor, beams, columns and other structures only is separated. On the other hand, if foreground / background separation processing by temporal correlation is performed on the image of FIG. 25 (b3) separated by spatial correlation from FIG. 25 (b1), it becomes as shown in FIG. 25 (b4). That is, a person in a moving audience seat is separated, and the background of only the structure of the floor, beams, columns, etc. of the seating seat is separated.

  That is, by performing foreground / background separation processing based on temporal correlation and foreground / background separation processing based on spatial correlation on the original image (FIG. 25 (b1)), the target background image is obtained regardless of the order of the separation processing. (FIG. 25 (b4)) can be obtained. That is, although the flowchart of FIG. 24B shows the foreground / background separation processing by spatial correlation after the foreground / background separation processing by temporal correlation, the order of the processing may be reversed. Furthermore, in the deviation detection and notification unit, foreground / background separation processing by temporal correlation and foreground / background separation processing by spatial correlation may be performed simultaneously.

  As a result, foreground / background separation processing by temporal correlation and foreground / background separation processing by spatial correlation in this gap detection / notification unit 5009, as shown in FIG. 25 (b4), for example, floor, beam, pillar, etc. A background of only structures is obtained. This background is a stable background image which has little temporal or spatial change and which is suitable for detecting a deviation of the camera installation state as shown in FIG. 24 (a).

  Therefore, when the installation adjustment of a plurality of cameras is completed in the stadium, an appropriate reference image can be obtained by the above-described image processing even in the situation where players and persons concerned with the hall are on the field. That is, there is no need to wait until no one is out of the field as in the prior art in order to obtain a reference image, and the efficiency of the installation work is improved. Also, by comparing the background of the current image obtained by the above-mentioned image processing with the background of the reference image, the camera can stably move even if there is an object moving within the shooting range at the time of shooting of the current image. It is possible to detect the deviation of

  Also, the background image obtained by the above-mentioned image processing is a distant background at a distance from the camera. Therefore, it is possible to accurately detect the influence of the camera angle variation as the image shift under the condition that the influence of the shift (parallax) of the translational direction of the camera with respect to the optical axis direction of the camera is small. The factor of the size of the camera as a shooting object disappears as the shooting target moves farther, and it can be treated as a point of only a pure position, so that it is possible to detect the influence of the camera's angular deviation more accurately.

  As described above, according to the first embodiment, the camera shift is achieved by using the background image obtained by applying the foreground / background separation processing to the captured image as the current image for comparison with the reference image. To detect Since the background image is not influenced by the foreground having a large variation factor, it is possible to more accurately detect the state change of the camera installation state (position, posture, etc.).

Second Embodiment
In the second embodiment, a form in which the current image is captured at another timing will be described. Specifically, in the first embodiment, the present image is periodically taken. However, in the second embodiment, the present image is taken when the camera has a vibration greater than or equal to a reference value. That is, the current image is taken at a timing when the camera installation state is likely to change. In addition, since it is the same as that of 1st Embodiment except imaging | photography timing, description is abbreviate | omitted.

  FIG. 26 is a flowchart of processing in the deviation detection annunciator in the second embodiment. First, in step S3801, an image is captured after calibration at the time of initial setting of the camera. Next, in step S3802, the foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the spatial correlation in the foreground separation unit 5001 with respect to the image captured at the time of initial setting of the camera. The details of the temporal correlation and foreground / background separation processing for separating the foreground and the background from the spatial correlation are the same as in the first embodiment. Then, the background image 5002 is stored in the reference background image storage unit 5008 as a reference image.

  Next, in step S3803, it is detected whether the vibration level applied to the camera is equal to or higher than a reference value. The sensor that detects this vibration is the external sensor 114 that configures each of the above-described sensor systems 110, and as described above, for example, information regarding the vibration can be obtained by a gyro sensor. Of course, the external sensor 114 may be configured of another sensor such as an acceleration sensor as long as it detects vibration. As this reference value, it is preferable to derive in advance a threshold for deviation that affects the image, and to adopt this value. Here, when a vibration equal to or greater than the reference value is detected, the current image is captured in step S3804. In step S3805, this image is subjected to foreground / background separation processing for separating the foreground and the background from the temporal correlation and the spatial correlation as described above in the foreground separation unit 5001, and the background image 5002 is compared as the current image. It is stored in the background image storage unit 5009.

  Next, in step S3806, the shift detection unit 5010 compares the reference image of the reference background image storage unit 5008 with the comparison current image of the comparison current background image storage unit 5009 to detect a shift. The comparison method uses a method using vectors between corresponding feature points, and other known techniques. If the deviation is larger than a predetermined reference value, it is determined that the installation state of the camera has changed, and alarm information is transmitted to the transmission unit 6120 to notify an alarm.

  As described above, according to the second embodiment, the current image is captured when the camera has a vibration equal to or greater than the reference value. With this configuration, it is possible to determine the presence or absence of a change in the camera installation state at a more appropriate timing. In addition, it is possible to notify the outside of the camera installation state by alarm.

Third Embodiment
In the third embodiment, a form of updating a reference image will be described. Specifically, in the first embodiment, a mode has been described in which a reference image based on a photographed image during installation processing is continuously used, but in the third embodiment, a difference in brightness (a change in brightness or more than a reference value) in image comparison The reference image is updated when the) is detected. In addition, since it is the same as that of 1st Embodiment except the update of a reference | standard image, description is abbreviate | omitted.

  FIG. 27 is a flowchart of processing in the deviation detection annunciator in the third embodiment. First, in step S3901, an image is captured after completion of calibration at the time of initial setting of the camera. Next, in step S3902, the foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the temporal correlation and the spatial correlation with the image captured at the time of this camera initial setting. The details of the temporal correlation and foreground / background separation processing for separating the foreground and the background from the spatial correlation are the same as in the first embodiment. Then, the background image storage unit 5008 stores the background image 5002 as a reference image. Next, in step S3903, the current image is captured. This timing may be at the time of activation or periodically as in the first embodiment, but may be any timing at which the operator notices a change in the environment and performs an operation.

  Next, in step S3904, the reference image and the current image are compared, and it is detected whether the change (difference) in luminance is equal to or greater than the reference value. Although this flowchart shows a configuration in which the luminance difference is detected from the image by comparing the images captured in step S3903 and step S3904, other configurations may be used as long as they are means for detecting a change in luminance. Good. For example, for example, a luminance sensor may be used for the external sensor 114 constituting each sensor system 110 to obtain information on a change in luminance. As the reference value regarding the luminance change, a limit value of the luminance change affecting the image is derived in advance, and this value is adopted. Here, when a change in luminance above the reference value is detected, foreground / background separation processing is performed on the current image in step S3905, and the background image is stored in the reference background image storage unit 5008 as a new reference image. Update and save past reference images. Then, the process is repeated to step S3903. If no change in luminance above the reference value is detected, this is the same as the embodiment described above, and thus the description will be omitted.

  As described above, according to the third embodiment, the reference image is updated when a change in luminance above the reference value is detected in the comparison between the reference image and the current image. With this configuration, even when there is a luminance change (such as a change in brightness due to a difference in time zone outdoors) that affects the image, an appropriate image comparison becomes possible, and a change in the camera installation state is suitably detected. It becomes possible.

Fourth Embodiment
In the fourth embodiment, an embodiment in which the fixation point of the camera is not fixed to one point will be described. The configuration other than the change of the fixation point is the same as that of the first embodiment, and thus the description is omitted.

  FIG. 28 is a flowchart of processing in the deviation detection annunciator in the fourth embodiment. First, in step S4001, information on a plurality of fixation points assumed in advance is input. Specifically, it is the pan / tilt angle on the platform of the camera, the shooting conditions of the camera (zoom value, aperture, shutter, gain, etc.), or information on the gaze point object as a target. Next, in step S4002, calibration is performed for each of the plurality of fixation points set above, and a subsequent image is captured as an image at the time of initial setting.

  In step S4003, the foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the temporal correlation and the spatial correlation, with respect to the image captured at the time of camera initialization. The details of the temporal correlation and foreground / background separation processing for separating the foreground and the background from the spatial correlation are the same as in the first embodiment. Then, the plurality of background images 5002 for each fixation point are stored in the reference background image storage unit 5008 as a plurality of reference images for each fixation point.

  Next, in step S4004, the current image is photographed at any of the predetermined fixation points. Next, in step S4005, the following operation is performed. The foreground separation unit 5001 performs foreground correlation processing to separate the foreground and the background from the temporal correlation and the spatial correlation in the foreground separation unit 5001, and compares the background image 5002 as the current image at the gaze point. It is stored in the background image storage unit 5009.

  In step S4006, a reference image corresponding to the same gaze point as the current shooting is selected. In step S4007, the shift detection unit 5010 detects a shift by comparing the reference image of the reference background image storage unit 5008 for the same selected gaze point with the comparison current image of the comparison current background image storage unit 5009. The subsequent steps are the same as the above-described embodiment and thus will not be described.

  As described above, according to the fourth embodiment, it is possible to apply the same change detection of the camera installation state as the above embodiment to a system in which a plurality of fixation points exist. In addition, the virtual viewpoint image can be easily generated regardless of the size of the devices constituting the system, such as the number of cameras 112, and the output resolution and output frame rate of the captured image.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  110a sensor system; 111a microphone; 112a camera; 113a camera head; 120a camera adapter; 180 switching hub; 190 end user terminal; 230 front end server; 250 database; 270 back end server; 290 time server; 310 control station; Camera operation UI

Claims (11)

A receiving unit configured to receive a captured image captured by the imaging device;
Separating means for separating a background image from the captured image received by the receiving means;
First storage means for storing a reference image based on a first captured image captured at a first time by the imaging device;
Second storage means for storing a current image based on a second captured image captured by the imaging device at a second time different from the first time;
A detection unit that detects a change in the state of the imaging device by comparing the reference image and the current image;
Have
The image processing apparatus according to claim 1, wherein the current image is a background image separated from the second captured image by the separation unit. The image processing apparatus according to claim 1, wherein the reference image is a background image separated from the first captured image by the separation unit. The image processing apparatus according to claim 1, wherein the detection unit includes an output unit that outputs an alarm when a change in state of the imaging device is larger than a predetermined reference value. The separation means is temporally correlated by comparison between a first reference background image generated based on one or more images captured by the imaging device prior to a captured image of the separation target and the captured image of the separation target And a spatial correlation based on a comparison between a second reference background image generated based on one or more images captured by an imaging device other than the imaging device and the captured image of the separation target. The image processing apparatus according to any one of claims 1 to 3, wherein the background image is separated from the pickup image of the separation target. It further comprises detection means for detecting the vibration of the imaging device,
The image processing apparatus according to any one of claims 1 to 4, wherein the second time is determined based on detection of a vibration equal to or more than a predetermined reference value by the detection unit. Luminance difference detection means for detecting a luminance difference between the reference image and the current image;
Updating means for updating the reference image based on the current image when a luminance difference equal to or greater than a predetermined threshold is detected by the luminance difference detection means;
The image processing apparatus according to any one of claims 1 to 5, further comprising: The imaging device is configured to be able to switch the fixation point,
The first storage unit stores a reference image for each gaze point;
When the fixation point of the image pickup apparatus is changed, the detection means reads out a reference image corresponding to the changed fixation point from the first storage means and compares it with the current image. The image processing apparatus according to any one of 1 to 6. The image processing apparatus according to any one of claims 1 to 7, wherein the imaging device is one of a plurality of imaging devices that capture a plurality of viewpoint images used to generate virtual viewpoint content. . A control method of an image processing apparatus for detecting a change in an installation state of an imaging device, comprising:
A receiving step of receiving a captured image captured by the imaging device;
A first storage step of storing, in a first storage unit, a reference image based on a first captured image captured at a first time by the imaging device;
A second storage step of separating a background image from a second captured image captured by the imaging device at a second time different from the first time and storing the background image as a current image in a second storage unit;
Detecting the state change of the imaging device by comparing the reference image and the current image;
A control method characterized by including. An image processing system for generating a virtual viewpoint image, comprising
A plurality of imaging devices that capture images from a plurality of different directions;
A plurality of image processing devices corresponding to each of the plurality of imaging devices, processing a captured image captured by the corresponding imaging device, and generating an image of a predetermined format;
An image generation device that generates a virtual viewpoint image based on a plurality of images of the predetermined format generated by the plurality of image processing devices;
Have
At least one of the plurality of image processing devices is
Receiving means for receiving a captured image captured by a corresponding imaging device;
Separating means for separating a background image from the captured image received by the receiving means;
First storage means for storing a reference image based on a first captured image captured at a first time by the imaging device;
Second storage means for storing a current image based on a second captured image captured by the imaging device at a second time different from the first time;
A detection unit that detects a change in the state of the imaging device by comparing the reference image and the current image;
Have
The image processing system according to claim 1, wherein the current image is a background image separated from the second captured image by the separation unit.   The program for functioning a computer as each means of the image processing apparatus in any one of Claims 1-8.

Images