JP2021077120A - Image processing system, controller and control method therefor, image transmission method and program - Google Patents

Abstract

To provide an image processing system, having a plurality of imaging apparatuses and a plurality of image processing apparatuses, which executes load distribution and transmits a picked-up image picked up at the same timing to the same image processor.SOLUTION: An image processing system (100) includes a plurality of camera adapters (120) corresponding to a plurality of cameras (112) and acquiring a plurality of images acquired by picking-up with the plurality of cameras (112) and a plurality of front end servers (131) acquiring a plurality of images transmitted from the plurality of camera adapters and executing image processing. Each of the plurality of camera adapters (120) transmits an image to a first front end server (131a) included in the plurality of front end servers on the basis of the fact that picking-up timing information given to the image is the first picking-up timing information and transmits the image to a second front end server (131b) included in the plurality of front end servers on the basis of the fact that the picking-up timing information given to the image is the second picking-up timing information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing system, a control device and its control method, an image transmission method, and a program.

Recently, a technique of installing a plurality of cameras at different positions, performing synchronous imaging from multiple viewpoints, and generating virtual viewpoint contents using a plurality of images obtained by the imaging has attracted attention. According to the technology for generating virtual viewpoint contents from a plurality of images, for example, highlight scenes of soccer and basketball can be viewed from various angles. Therefore, it is possible to give the user a high sense of presence as compared with a normal image.

To generate and view virtual viewpoint content based on multiple images, the images captured by multiple cameras are aggregated in an image processing unit such as a server, and the image processing unit performs processing such as 3D model generation and rendering. , It is realized by transmitting to the user terminal. However, in order to perform processing such as three-dimensional model generation and rendering using images captured by a plurality of cameras, a high ability is required for an image processing unit such as a server. Therefore, it is conceivable to perform load balancing using a plurality of servers.

In order to perform load balancing, a technique for appropriately distributing a plurality of images to a plurality of servers is required. Patent Document 1 describes that data access to the load balancer is distributed based on the result of calculating the Hash value. Patent Document 2 describes that load balancing is performed by adding the current time to a parameter for determining a server to which data is transmitted.

Japanese Unexamined Patent Publication No. 2003-131961 Japanese Unexamined Patent Publication No. 2014-098975

In the generation of virtual images, it is desirable to aggregate a plurality of captured images captured at the same timing by synchronous imaging into one image processing unit. However, the load distribution method described in Patent Documents 1 and 2 has a problem that it is not possible to sort a plurality of captured images captured at the same timing into one image processing unit.

According to one aspect of the present invention, in an image processing system including a plurality of image pickup devices and a plurality of image processing devices, load distribution is performed and captured images captured at the same timing are transmitted to the same image processing device. The technology that makes it possible is provided.

The image processing system according to one aspect of the present invention is
A plurality of control devices corresponding to a plurality of image pickup devices and acquiring a plurality of images acquired by being imaged by the plurality of image pickup devices, and a plurality of control devices.
A plurality of image processing devices for acquiring the plurality of images transmitted from the plurality of control devices, and a plurality of image processing devices.
Have,
The plurality of control devices transmit an image to a first image processing device included in the plurality of image processing devices based on the imaging timing information given to the image being the first image processing timing information. Based on the imaging timing information given to the image being the second imaging timing information, the image is transmitted to the second image processing apparatus included in the plurality of image processing apparatus.

According to the present invention, in an image processing system provided with a plurality of image pickup devices and a plurality of image processing devices, it is possible to perform load distribution and transmit the captured images captured at the same timing to the same image processing device. ..

A block diagram showing a configuration example of an image processing system. A block diagram showing an example of a functional configuration of a camera adapter. A block diagram showing a functional configuration example of the front-end server. A block diagram showing a functional configuration example of the data input control unit of the front-end server. A flowchart that describes the entire workflow. A flowchart explaining the workflow before installing the equipment. A flowchart explaining the workflow when installing the equipment. The sequence diagram explaining the process of calibration at the time of installation. The sequence diagram explaining the imaging start process. The sequence diagram explaining the imaging start process. A sequence diagram illustrating a process of generating three-dimensional model information. The figure which shows the data format which a camera adapter outputs. The flowchart explaining the operation of the camera adapter of 1st Embodiment. The figure explaining the setting of the address by the camera adapter of 1st Embodiment. The figure explaining the setting of the address by the camera adapter of 1st Embodiment. A flowchart illustrating a file generation process. A block diagram showing a hardware configuration example of a camera adapter. The flowchart explaining the operation of the camera adapter of 2nd Embodiment. The figure explaining the setting of the address by the camera adapter of the 2nd Embodiment.

<First Embodiment>
A system for imaging and collecting sound using a plurality of cameras and microphones installed in facilities such as a stadium and a concert hall will be described with reference to the system configuration diagram of FIG. The image processing system 100 of the first embodiment includes sensor systems 110a to 110z, an image computing server 130, a controller 140, a switching hub 180, and an end user terminal 150. The image processing system 100 generates a virtual viewpoint image based on images obtained from a plurality of cameras (sensor systems 110a to 110z).

The controller 140 has a control station 141 and a virtual camera operation UI 142. The control station 141 manages the operating state and controls the parameter setting for each block constituting the image processing system 100 through the networks 190a to 190d, 180a, 180b, and 170a to 170y. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard (registered trademark), or may be configured by combining an interconnect Infiniband, an industrial Ethernet, or the like. Further, the network is not limited to these, and may be another type of network.

Next, an operation of transmitting 26 sets of images and sounds acquired by the sensor systems 110a to 110z from the sensor system 110z to the image computing server 130 will be described. In the image processing system 100 of the first embodiment, the sensor systems 110a to 110z are connected by a daisy chain.

In the first embodiment, unless otherwise specified, the 26 sets of systems from the sensor system 110a to the sensor system 110z are referred to as the sensor system 110 without distinction. Similarly, the devices in each sensor system 110 are not distinguished unless otherwise specified, and are described as a microphone 111, a camera 112, a pan head 113, an external sensor 114, and a camera adapter 120. The case where the sensor system has 26 sets is described, but this is just an example, and the number of sensor systems is not limited to this. Further, in the first embodiment, unless otherwise specified, the word "image" will be described as including the concepts of moving images and still images. That is, the image processing system 100 of the first embodiment can process both still images and moving images.

Further, in the first embodiment, an example in which the virtual viewpoint image and the virtual viewpoint sound are included in the virtual viewpoint content provided by the image processing system 100 will be mainly described, but the present invention is not limited to this. For example, the virtual viewpoint content does not have to include audio. Further, for example, the sound included in the virtual viewpoint content may be the sound picked up by the microphone closest to the virtual viewpoint. Further, in the first embodiment, for the sake of simplification of the explanation, the description about the sound is partially omitted, but basically both the image and the sound are processed.

The image processing system 100 has a plurality of cameras for capturing an image of a subject from a plurality of directions. In the present embodiment, the sensor systems 110a to 110z each have one camera 112a to 112z as an imaging device. One sensor system 110 may be connected to a plurality of cameras 112. The plurality of sensor systems 110 are connected to each other by a daisy chain. It is said that this connection form has the effect of reducing the number of connection cables and labor saving in wiring work in increasing the resolution of captured images to 4K or 8K and increasing the capacity of image data due to the increase in frame rate. I will specify it here. The connection form of the plurality of sensor systems is not limited to this. For example, each of the sensor systems 110a to 110z is connected to the switching hub 180, and data is transmitted / received between the sensor systems 110 via the switching hub 180. This may be a star-type network configuration.

Further, FIG. 1 shows a configuration in which all of the sensor systems 110a to 110z are cascade-connected so as to form a daisy chain, but the present invention is not limited to this. For example, a plurality of sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy-chained in the divided group units. Further, in this case, the camera adapter 120 which is the terminal of each group may be connected to the switching hub to input the image to the image computing server 130. Such a configuration is particularly effective in a stadium. For example, a stadium may be composed of a plurality of floors, and a sensor system 110 may be installed on each floor. In this case, the sensor system can be grouped for each floor or every half lap of the stadium, and input to the image computing server 130 can be performed. As a result, it is possible to simplify the installation and make the system flexible even in a place where wiring for connecting all the sensor systems 110 with one daisy chain is difficult.

Further, the control of image processing in the image computing server 130 is switched depending on whether the number of camera adapters 120 connected in a daisy chain and inputting images to the image computing server 130 is one or two or more. .. That is, the control is switched depending on whether or not the sensor system 110 is divided into a plurality of groups. When there is one camera adapter 120 for inputting an image to the image computing server 130, an image of the entire circumference of the stadium is generated while performing image transmission with one group of daisy chain connections. Therefore, the timing at which the image data of the entire circumference is gathered in the image computing server 130 is synchronized. That is, if the sensor system 110 is not divided into groups, synchronization can be achieved.

However, when there are a plurality of camera adapters 120 for inputting images to the image computing server 130 (sensor systems 110 are divided into groups), the delay time differs depending on the daisy chain lane (route) in each group. Can be considered. Therefore, it is clearly stated that it is necessary to perform the subsequent image processing while checking the collection of the image data by the synchronization control in which the image computing server 130 waits until the image data of the entire circumference is prepared and synchronizes.

Next, the sensor system 110 will be described. In the first embodiment, the sensor system 110 includes a microphone 111, a camera 112, a pan head 113, an external sensor 114, and a camera adapter 120. The configuration is not limited to this, and it is sufficient to have at least one camera adapter 120 and one camera 112 or one microphone 111. Further, for example, the sensor system 110 may be composed of one camera adapter 120 and a plurality of cameras 112, or may be composed of one camera 112 and a plurality of camera adapters 120. That is, the plurality of cameras 112 and the plurality of camera adapters 120 in the image processing system 100 correspond to each other by N to M (N and M are both integers of 1 or more).

Further, the sensor system 110 may include devices other than the microphone 111, the camera 112, the pan head 113, and the camera adapter 120. Further, the camera 112 and the camera adapter 120 may be integrally configured. Further, the front-end servers 131a to 131b may have at least a part of the functions of the camera adapter 120. In the first embodiment, it is assumed that the sensor systems 110a to 110z have the same configuration, but not all of the sensor systems 110a to 110z are limited to the same configuration, and each sensor system 110 has a similar configuration. It may have a different configuration.

The sound picked up by the microphone 111a and the image captured by the camera 112a are transmitted to the camera adapter 120b of the sensor system 110b through the daisy chain 170a after the image processing described later is performed by the camera adapter 120a. .. Similarly, the sensor system 110b transmits the picked-up sound and the captured image to the sensor system 110c together with the image and the sound acquired from the sensor system 110a. By continuing such an operation, the images and sounds acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z to the switching hub 180 via the network 180b, and then transmitted to the image computing server 130.

In the present embodiment, the camera 112 and the camera adapter 120 are separated from each other, but the camera 112 and the camera adapter 120 may be integrated in the same housing. In that case, the microphone 111 may be built in the integrated camera 112 or may be connected to the outside of the camera 112.

Next, the configuration and operation of the image computing server 130 will be described. The image computing server 130 of the present embodiment processes the data acquired from the sensor system 110z. The image computing server 130 includes front-end servers 131a to 131b, a database 132 (hereinafter, also referred to as DB), a back-end server 133, and a time server 134. In the first embodiment, unless otherwise specified, the front-end servers 131a to 131b are referred to as the front-end server 131 without distinction. Further, although the number of front-end servers 131 is described as two, this is just an example, and the number is not limited to this. That is, the number of front-end servers 131 may be three or more.

The time server 134 has a function of distributing the time and synchronization signals, and distributes the time and synchronization signals to the sensor systems 110a to 110z via the switching hub 180. The camera adapters 120a to 120z that have received the time and synchronization signals genlock the cameras 112a to 112z based on the time and synchronization signals to perform image frame synchronization. That is, the time server 134 synchronizes the imaging timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured (synchronously captured) at the same timing, so that deterioration of the quality of the virtual viewpoint image due to a deviation in the imaging timing can be suppressed. In the present embodiment, the time server 134 manages the time synchronization of the plurality of cameras 112, but the present invention is not limited to this, and each camera 112 or each camera adapter 120 independently performs the processing for time synchronization. You may go there.

The front-end server 131 is an example of an image processing device that processes a plurality of images captured by a plurality of cameras 112. The image processing system 100 has a plurality of front-end servers. In the present embodiment, each of the plurality of front-end servers 131 reconstructs the segmented transmission packets of the image and the sound acquired from the sensor system 110z to convert the data format. The front-end server 131 writes the image and sound thus obtained in the database 132 according to the camera identifier, the data type, and the frame number. Details of the front-end server 131 will be described later. Further, in the present embodiment, the images captured at the same timing among the images obtained by the synchronous imaging of the plurality of cameras 112 are delivered to the same front-end server 131 among the plurality of front-end servers 131. ing. This configuration improves the efficiency of image processing in the front-end server 131. The back-end server 133 receives the designation of the viewpoint from the virtual camera operation UI 142, reads the corresponding image and audio data from the database 132 based on the received viewpoint, performs rendering processing, and generates a virtual viewpoint image.

The configuration of the image computing server 130 is not limited to this. For example, at least two of the front-end server 131, the database 132, and the back-end server 133 may be integrally configured. Further, at least one of the front-end server 131, the database 132, and the back-end server 133 may be included. Further, a device other than the above device may be included at an arbitrary position in the image computing server 130. Further, the end user terminal 150 or the virtual camera operation UI 142 may have at least a part of the functions of the image computing server 130.

The rendered image is transmitted from the back-end server 133 to the end-user terminal 150. In this way, the user who operates the end user terminal 150 can view images and listen to audio according to the designation of the viewpoint. That is, the back-end server 133 generates virtual viewpoint contents based on the captured images (multi-viewpoint images) captured by the plurality of cameras 112 and the viewpoint information. More specifically, the back-end server 133 uses, for example, a virtual viewpoint based on image data of a predetermined area extracted from images captured by a plurality of cameras 112 by a plurality of camera adapters 120 and a viewpoint designated by a user operation. Generate content. Then, the back-end server 133 provides the generated virtual viewpoint content to the end user terminal 150. Details of extraction of a predetermined area by the camera adapter 120 will be described later. The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when a subject is imaged from a virtual viewpoint. In other words, the virtual viewpoint image can be said to be an image representing the appearance at the specified viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily specified by the user. In addition, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to a viewpoint automatically designated by the device are also included in the virtual viewpoint image.

In the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described, but the audio data may not necessarily be included. In addition, the back-end server 133 displays the virtual viewpoint image in H. It may be compressed and encoded by a standard technique represented by 264 or HEVC, and then transmitted to the end user terminal 150 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 150 without compression. In particular, the former that performs compression coding assumes a smartphone or tablet as the end user terminal 150. The latter envisions a display capable of displaying uncompressed images. That is, it is clearly stated that the image format can be switched according to the type of the end user terminal 150. Further, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or other transmission method may be used.

As described above, the image processing system 100 has three functional domains, that is, a video acquisition domain, a data storage domain, and a video generation domain. The video acquisition domain includes sensor systems 110a-110z. The data storage domain includes the database 132, the front-end servers 131a-131b and the back-end server 133. The video generation domain includes a virtual camera operation UI 142 and an end user terminal 150. Not limited to this configuration, for example, the virtual camera operation UI 142 can directly acquire an image from the sensor systems 110a to 110z. However, in the present embodiment, the method of arranging the data storage function in the middle is adopted instead of the method of directly acquiring the image from the sensor systems 110a to 110z. Specifically, the front-end servers 131a to 131b convert the image data and audio data generated by the sensor systems 110a to 110z and the meta information of those data into the common schema and data type of the database 132. As a result, even if the camera 112 of the sensor system 110 is changed to a camera of another model, the front-end servers 131a to 131b can absorb the changed difference and register it in the database 132. As a result, when the camera 112 is changed to another model camera, the possibility that the virtual camera operation UI 142 does not operate properly can be reduced.

Further, the virtual camera operation UI 142 is configured to access the database 132 via the back-end server 133 instead of directly accessing the database 132. The back-end server 133 performs common processing related to the image generation processing, and the virtual camera operation UI 142 performs the difference portion of the application related to the operation UI. As a result, in the development of the virtual camera operation UI 142, it is possible to focus on the development of the UI function requirements for the UI operation device and the UI for operating the virtual viewpoint image to be generated. Further, the back-end server 133 can add or delete common processing related to the image generation processing in response to the request of the virtual camera operation UI 142. This makes it possible to flexibly respond to the demands of the virtual camera operation UI 142.

As described above, in the image processing system 100, the back-end server 133 generates a virtual viewpoint image based on the image data based on the images taken by the plurality of cameras 112 for capturing the subject from a plurality of directions. The image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be logically configured.

Next, the functional configuration of each node (camera adapter 120, front-end server 131, database 132, back-end server 133, virtual camera operation UI 142, end-user terminal 150) in the system shown in FIG. 1 will be described.

FIG. 2 is a block diagram showing a functional configuration example of the camera adapter 120. The functional configuration of the camera adapter 120 according to the first embodiment will be described with reference to FIG. The camera adapter 120 includes a network adapter 210, a transmission unit 220, an image processing unit 230, and an external device control unit 240.

The network adapter 210 includes a data transmission / reception unit 211 and a time control unit 212.

The data transmission / reception unit 211 performs data communication with other camera adapters 120, front-end servers 131a and 131b, time server 134, and control station 141 via the daisy chain 170, network 135, and network 190a. For example, the data transmission / reception unit 211 outputs the foreground image and the background image separated by the foreground background separation unit 231 from the image captured by the camera 112 to another camera adapter 120. The output destination camera adapter 120 is, for example, the next camera adapter 120 in the camera adapter 120 in the image processing system 100 in a predetermined order according to the processing of the data routing processing unit 222. By outputting the foreground image and the background image from each camera adapter 120, a virtual viewpoint image is generated based on the foreground image and the background image captured from a plurality of viewpoints. There may be a camera adapter 120 that outputs a foreground image separated from the captured image and does not output a background image.

The time control unit 212 conforms to, for example, the Ordinary Clock of the IEEE1588 standard, and performs a function of storing a time stamp of data sent and received to and from the time server 134 and time synchronization with the time server 134. It is not limited to IEEE1588, and time synchronization with a time server may be realized by another EtherAVB standard or an original protocol. In the present embodiment, the NIC (Network Interface Card) is used as the network adapter 210, but the present invention is not limited to the NIC, and other similar interfaces may be used. In addition, IEEE1588 has been updated as a standard such as IEEE1588-2002 and IEEE1588-2008, and the latter is also called PTPv2 (Precision Time Protocol Version 2).

The transmission unit 220 has a function of controlling the transmission of data to the switching hub 180 and the like via the network adapter 210. The functional unit included in the transmission unit 220 will be described below.

The data compression / decompression unit 221 has a function of compressing data transmitted / received via the data transmission / reception unit 211 by applying a predetermined compression method, compression rate, and frame rate, and a function of decompressing the compressed data. have.

The data routing processing unit 222 uses the data held by the data routing information holding unit 225, which will be described later, to determine the routing destination of the data received by the data transmitting / receiving unit 211 and the data processed by the image processing unit 230. Further, the data routing processing unit 222 has a function of transmitting data to the determined routing destination. As the routing destination, it is preferable to use a camera adapter 120 corresponding to the camera 112 focused on the same gazing point in order to perform image processing. This is because the image frame correlation between the cameras 112 focused on the same gazing point is high. The order of the camera adapters 120 that output the foreground image and the background image in the relay format in the image processing system 100 is determined according to the determination by the data routing processing unit 222 of each of the plurality of camera adapters 120.

The time synchronization control unit 223 conforms to PTP (Precision Time Protocol) of the IEEE1588 standard, and has a function of performing processing related to time synchronization with the time server 134. The time may be synchronized by using another similar protocol instead of limiting to PTP.

The image / audio transmission processing unit 224 has a function of creating a message for transferring image data or audio data to another camera adapter 120 or a front-end server 131 via the data transmission / reception unit 211. The message includes image data or audio data, and meta information of each data. The meta information of the present embodiment includes a time code or sequence number at the time of image capturing or sound sampling, a data type, and an identifier indicating an individual of the camera 112 or the microphone 111. The transmitted image data or audio data may be data-compressed by the data compression / decompression unit 221. Further, the image / audio transmission processing unit 224 receives a message from another camera adapter 120 via the data transmission / reception unit 211. Then, the image / audio transmission processing unit 224 restores the data information fragmented to the packet size defined by the transmission protocol into the image data or the audio data according to the data type included in the message. If the data is compressed when the data is restored, the data compression / decompression unit 221 performs the decompression process.

The data routing information holding unit 225 has a function of holding address information for determining a transmission destination of data transmitted / received by the data transmitting / receiving unit 211. The routing method will be described later.

The image processing unit 230 has a function of processing the image data captured by the camera 112 and the image data received from the other camera adapter 120 under the control of the camera control unit 241. The functional unit included in the image processing unit 230 will be described below.

The foreground background separation unit 231 has a function of separating the image data captured by the camera 112 into a foreground image and a background image. That is, the foreground background separation unit 231 of each of the plurality of camera adapters 120 extracts a predetermined region from the image captured by the corresponding camera 112 among the plurality of cameras 112. The predetermined area is, for example, a foreground image obtained as a result of object detection on a captured image. By extracting the predetermined region, the foreground background separation unit 231 separates the captured image into the foreground image and the background image. The object is, for example, a person. However, the object may be a specific person (player, manager, and / or referee, etc.), or may be an object having a predetermined image pattern, such as a ball or a goal. Further, a moving object may be detected as an object. By separating and processing the foreground image including an important object such as a person and the background area not including such an object, the image of the part corresponding to the above object of the virtual viewpoint image generated in the image processing system 100. The quality of the object can be improved. Further, by separating the foreground and the background from each of the plurality of camera adapters 120, the load on the image processing system 100 provided with the plurality of cameras 112 can be distributed. The predetermined area is not limited to the foreground image, and may be, for example, a background image.

The three-dimensional model information generation unit 232 uses the foreground image separated by the foreground background separation unit 231 and the foreground image received from another camera adapter 120, and uses, for example, the principle of a stereo camera to provide image information related to the three-dimensional model. Has a function to generate.

The calibration control unit 233 has a function of acquiring image data required for calibration from the camera 112 via the camera control unit 241 and transmitting the image data to the front-end server 131 that performs arithmetic processing related to the calibration. .. In the present embodiment, the arithmetic processing related to the calibration is performed by the front-end server 131, but the node that performs the arithmetic processing is not limited to the front-end server 131. For example, arithmetic processing may be performed at another node such as the control station 141 or the camera adapter 120 (including another camera adapter 120). Further, the calibration control unit 233 has a function of performing calibration (dynamic calibration) during imaging according to preset parameters for the image data acquired from the camera 112 via the camera control unit 241. doing.

The external device control unit 240 has a function of controlling a device connected to the camera adapter 120. The functional unit included in the external device control unit 240 will be described below.

The camera control unit 241 is connected to the camera 112 and has a function of controlling the camera 112, acquiring a captured image, providing a synchronization signal, setting a time, and the like. For control of the camera 112, for example, setting and reference of imaging parameters (number of pixels, color depth, frame rate, white balance setting, etc.), state of the camera 112 (imaging, stopped, synchronizing, error, etc.) Acquisition, start and stop of imaging, focus adjustment, etc. In the present embodiment, the focus is adjusted via the camera 112, but when a removable lens is attached to the camera 112, the camera adapter 120 is connected to the lens to directly adjust the lens. May be good. Further, the camera adapter 120 may perform lens adjustment such as zooming via the camera 112.

The synchronization signal is provided by the time synchronization control unit 223 using the time synchronized with the time server 134 and providing the imaging timing (control clock) to the camera 112. The time setting is performed by the time synchronization control unit 223 providing the time synchronized with the time server 134 with a time code conforming to, for example, the SMPTE 12M format. As a result, the provided time code is added to the image data received from the camera 112 (details will be described later with reference to FIG. 9A). The time code format is not limited to SMPTE12M, and may be another format. Further, the camera control unit 241 may not provide the time code to the camera 112, but may assign the time code to the image data received from the camera 112 (details will be described later with reference to FIG. 9B).

The microphone control unit 242 is connected to the microphone 111 and has a function of controlling the microphone 111, starting and stopping sound collection, acquiring sound collected voice data, and the like. The control of the microphone 111 is, for example, gain adjustment, state acquisition, and the like. Further, similarly to the camera control unit 241, the microphone control unit 242 provides the microphone 111 with the timing and time code for voice sampling. As the clock information that is the timing of voice sampling, the time information from the time server 134 is converted into, for example, a 48 KHz word clock and supplied to the microphone 111.

The pan head control unit 243 has a function of connecting to the pan head 113 and controlling the pan head 113. Control of the pan head 113 includes, for example, pan / tilt control and state acquisition.

The sensor control unit 244 has a function of connecting to the external sensor 114 and acquiring sensor information sensed by the external sensor 114. For example, when a gyro sensor is used as the external sensor 114, information representing vibration can be acquired. Then, using the vibration information acquired by the sensor control unit 244, the image processing unit 230 can generate an image in which vibration is suppressed prior to the processing by the foreground background separation unit 231. The vibration information is used, for example, when the image data of an 8K camera is cut out in a size smaller than the original 8K size in consideration of the vibration information and aligned with the image of the adjacent camera 112. .. As a result, even if the skeleton vibration of the building propagates to each camera at different frequencies, the alignment is performed by this function provided in the camera adapter 120. As a result, electronically vibration-proof image data can be generated, and the effect of reducing the processing load of alignment for the number of cameras 112 in the image computing server 130 can be obtained. The sensor of the sensor system 110 is not limited to the external sensor 114, and the same effect can be obtained even if the sensor is built in the camera adapter 120.

FIG. 3 is a block diagram showing a functional configuration example of the front-end server 131. The control unit 310 is composed of a CPU, a DRAM, a storage medium such as an HDD or a NAND memory that stores program data and various data, and hardware such as Ethernet (registered trademark). The control unit 310 controls each functional unit of the front-end server 131 and the entire system of the front-end server 131. Further, the control unit 310 performs mode control to switch operation modes such as a calibration operation, a preparatory operation before imaging, and an operation during imaging. Further, the control unit 310 receives a control instruction from the control station 141 through Ethernet (registered trademark), switches each mode, inputs / outputs data, and the like. Further, the control unit 310 also acquires stadium CAD data (stadium shape data) from the control station 141 through the network, and transmits the stadium CAD data to the CAD data storage unit 335 and the non-imaging data file generation unit 385. The stadium CAD data (stadium shape data) in the present embodiment is three-dimensional data indicating the shape of the stadium, and may be any data representing a mesh model or other three-dimensional shape, and is not limited to the CAD format.

The data input control unit 320 is network-connected to the camera adapter 120 via a communication path such as Ethernet (registered trademark) and a switching hub 180. The data input control unit 320 acquires a foreground image, a background image, a three-dimensional model of a subject, audio data, and camera calibration captured image data from the camera adapter 120 through a network. Here, the foreground image is image data based on the foreground region of the captured image for generating the virtual viewpoint image, and the background image is image data based on the background region of the captured image. As described above, the camera adapter 120 specifies the foreground area and the background area according to the result of the detection processing of a predetermined object for the image captured by the camera 112, and generates the foreground image and the background image. The predetermined object is, for example, a person. The predetermined object may be a specific person (player, manager, and / or referee, etc.). Further, the predetermined object may include an object having a predetermined image pattern, such as a ball or a goal. Further, a moving object may be detected as a predetermined object.

Further, the data input control unit 320 transmits the foreground image and the background image acquired from the camera adapter 120 to the data synchronization unit 330, and transmits the camera calibration captured image data acquired from the camera adapter 120 to the calibration unit 340. Further, the data input control unit 320 has a function of performing compression / decompression of received data, data routing processing, and the like. Further, both the control unit 310 and the data input control unit 320 have a communication function by a network such as Ethernet (registered trademark), but the communication function may be shared by these. In that case, a method may be used in which the data input control unit 320 receives the instruction by the control command from the control station 141 or the stadium CAD data and sends it to the control unit 310.

The data synchronization unit 330 temporarily stores the data acquired from the camera adapter 120 on the DRAM, and buffers the foreground image, the background image, the audio data, and the three-dimensional model data until they are prepared. The foreground image, background image, audio data, and three-dimensional model data are collectively referred to as imaging data hereafter. Meta information such as routing information, time code information (time information), and camera identifier is added to the captured data, and the data synchronization unit 330 confirms the data attributes based on this meta information. As a result, the data synchronization unit 330 determines that the data is at the same time and confirms that the data is complete. This is because the reception order of network packets is not guaranteed for the data transferred from each camera adapter 120 by the network, and it is necessary to buffer until the data necessary for file generation is prepared. In the present embodiment, distributed processing is realized by a plurality of front-end servers (image processing devices), and data at the same time is distributed to each front-end server, so that the processing efficiency of image processing in the front-end server is improved. ..

When the data is prepared, the data synchronization unit 330 transmits the foreground image and the background image to the image processing unit 350, the three-dimensional model data to the three-dimensional model coupling unit 360, and the audio data to the imaging data file generation unit 380. The data prepared here is data necessary for file generation in the imaging data file generation unit 380, which will be described later. Further, the background image may be captured at a frame rate different from that of the foreground image. For example, when the frame rate of the background image is 1 fps, one background image is acquired every second. Therefore, for the time when the background image is not acquired, all the data may be collected without the background image. .. Further, the data synchronization unit 330 notifies the database 132 of information indicating that the data cannot be collected when the data is not available even after the lapse of a predetermined time. Then, when the database 132 in the subsequent stage stores the data, it stores the information indicating the lack of data together with the camera number and the frame number. As a result, it is possible to automatically notify before rendering whether or not a desired image can be formed from the captured image of the camera 112 in which data is collected in response to a viewpoint instruction from the virtual camera operation UI 142 to the back-end server 133. .. As a result, the visual load on the operator of the virtual camera operation UI 142 can be reduced.

The CAD data storage unit 335 stores the three-dimensional data indicating the stadium shape received from the control unit 310 in a storage medium such as a DRAM, an HDD, or a NAND memory. Then, the stadium shape data saved when the request for the stadium shape data is received is transmitted to the image combining unit 370.

The calibration unit 340 performs a camera calibration operation and sends the camera parameters obtained by the calibration to the non-imaging data file generation unit 385, which will be described later. At the same time, the camera parameters are also held in its own storage area, and the camera parameter information is provided to the three-dimensional model coupling unit 360 described later.

The image processing unit 350 performs processing such as matching of color and brightness values between cameras, development processing when RAW image data is input, and correction of camera lens distortion for the foreground image and background image. .. Then, the image-processed foreground image is transmitted to the imaging data file generation unit 380, and the background image is transmitted to the image combining unit 370.

The three-dimensional model coupling unit 360 combines the three-dimensional model data at the same time acquired from the camera adapter 120 using the camera parameters generated by the calibration unit 340. Then, using a method called VisualHull, three-dimensional model data of the foreground image in the entire stadium is generated. The generated three-dimensional model is transmitted to the imaging data file generation unit 380.

The image combining unit 370 acquires a background image from the image processing unit 350, acquires three-dimensional shape data (stadium shape data) of the stadium from the CAD data storage unit 335, and a background image with respect to the coordinates of the acquired three-dimensional shape data of the stadium. Identify the location of. When the position of each of the background images with respect to the coordinates of the three-dimensional shape data of the stadium can be specified, the background images are combined into one background image. The back-end server 133 may create the three-dimensional shape data of the background image.

The imaging data file generation unit 380 is combined with audio data from the data synchronization unit 330, foreground images from the image processing unit 350, three-dimensional model data from the three-dimensional model coupling unit 360, and three-dimensional shape from the image coupling unit 370. Get the background image. Then, these acquired data are output to the DB access control unit 390. Here, the imaging data file generation unit 380 outputs these data in association with each other based on the respective time information. However, some of these data may be associated and output. For example, the imaging data file generation unit 380 outputs the foreground image and the background image in association with each other based on the time information of the foreground image and the time information of the background image. Further, for example, the imaging data file generation unit 380 associates the foreground image, the background image, and the three-dimensional model data with each other based on the time information of the foreground image, the time information of the background image, and the time information of the three-dimensional model data. Output. The imaging data file generation unit 380 may file and output the associated data for each type of data, or collectively output a plurality of types of data as a file for each time indicated by the time information. May be good. By outputting the imaged data associated in this way to the database 132 by the DB access control unit 390, the back-end server 133 can generate a virtual viewpoint image from the foreground image and the background image corresponding to the time information.

When the frame rates of the foreground image and the background image acquired by the data input control unit 320 are different, it is difficult for the imaging data file generation unit 380 to always output the foreground image and the background image at the same time in association with each other. Therefore, the imaging data file generation unit 380 outputs the time information of the foreground image in association with the background image having the time information having a relationship based on a predetermined rule. Here, the background image having the time information of the foreground image and the time information having a relationship based on a predetermined rule is, for example, the time closest to the time information of the foreground image among the background images acquired by the imaging data file generation unit 380. It is a background image having information. By associating the foreground image with the background image based on a predetermined rule in this way, even if the frame rates of the foreground image and the background image are different, the virtual viewpoint image can be obtained from the foreground image and the background image captured at close times. Can be generated.

The method of associating the foreground image with the background image is not limited to the above. For example, the background image having the time information of the foreground image and the time information having a relationship based on a predetermined rule is the acquired background image and the background image having the time information corresponding to the time before the foreground image. It may be a background image having the time information closest to the time information of the foreground image. By communicating with each other, the plurality of front-end servers 131 can be selected including the background image processed by the other front-end servers. For example, the front-end server 131a (131b) selects a background image having the time information closest to the time information of the foreground image from the background image processed by itself and the background image processed by the front-end server 131b (131a). You may choose. According to this method, the associated foreground image and the background image can be output with low delay without waiting for the acquisition of the background image having a frame rate lower than that of the foreground image. Further, the background image having the time information of the foreground image and the time information having a relationship based on a predetermined rule is the acquired background image and among the background images having the time information corresponding to the time after the foreground image. It may be a background image having the time information closest to the time information of the foreground image.

The non-imaging data file generation unit 385 acquires camera parameters from the calibration unit 340 and three-dimensional shape data of the stadium from the control unit 310, shapes the stadium according to the file format, and then transmits the data to the DB access control unit 390. The camera parameters or stadium shape data, which are the data input to the non-imaging data file generation unit 385, are individually formed according to the file format. That is, when the non-imaging data file generation unit 385 receives either of the data, the non-imaging data file generation unit 385 individually transmits the data to the DB access control unit 390.

The DB access control unit 390 is connected to the database 132 so that high-speed communication is possible by InfiniBand or the like. Then, the files received from the imaging data file generation unit 380 and the non-imaging data file generation unit 385 are transmitted to the database 132. In the present embodiment, the imaging data associated with the imaging data file generation unit 380 based on the time information is sent to the database 132, which is a storage device connected to the front-end server 131 via the network, via the DB access control unit 390. Is output. However, the output destination of the associated imaging data is not limited to this. For example, the front-end server 131 outputs the imaging data associated based on the time information to the back-end server 133, which is an image generation device that is connected to the front-end server 131 via a network and generates a virtual viewpoint image. You may. Further, it may be output to both the database 132 and the back-end server 133.

Further, in the present embodiment, the front-end server 131 associates the foreground image with the background image, but the present invention is not limited to this, and the database 132 may associate the foreground image and the background image. For example, the database 132 acquires a foreground image and a background image having time information from the front-end server 131. Then, the database 132 may associate the foreground image and the background image with each other based on the time information of the foreground image and the time information of the background image, and output the image to the storage unit included in the database 132.

The functional configuration of the data input control unit 320 of the front-end server 131 will be described with reference to the block diagram of FIG. The data input control unit 320 includes a server network adapter 410, a server transmission unit 420, and a server image processing unit 430.

The server network adapter 410 has a server data receiving unit 411 and has a function of receiving data transmitted from the camera adapter 120.

The server transmission unit 420 has a function of processing the data received from the server data reception unit 411, and is composed of the following functional units. The server data decompression unit 421 has a function of decompressing the compressed data. The server data routing processing unit 422 determines a data transfer destination based on routing information such as an address held by the server data routing information holding unit 424, which will be described later, and transfers the data received from the server data receiving unit 411. The server image / audio transmission processing unit 423 receives a message from the camera adapter 120 via the server data receiving unit 411, and restores fragmented data into image data or audio data according to the data type included in the message. .. If the restored image data or audio data is compressed, the server data decompression unit 421 performs decompression processing. The server data routing information holding unit 424 has a function of holding address information for determining the transmission destination of the data received by the server data receiving unit 411.

The server image processing unit 430 has a function of performing processing related to image data or audio data received from the camera adapter 120. The processing content is, for example, attribute information of the camera number, the imaging time of the image frame, the image size, the image format, and the coordinates of the image according to the data entity (foreground image, background image, and three-dimensional model information) of the image data. It is a shaping process to the format to which etc. is given.

Next, the workflow in this embodiment will be described with reference to the flowchart of FIG. In this embodiment, a workflow in which a plurality of cameras 112 and microphones 111 are installed in a facility such as a stadium or a concert hall to perform imaging will be described. Before the start of the process shown in FIG. 5, the operator (user) who installs or operates the image processing system 100 collects necessary information (preliminary information) before the installation and formulates a plan. Further, it is assumed that the operator has installed the equipment in the target facility before the start of the process shown in FIG.

In S500 (pre-installation processing), the control station 141 of the controller 140 receives a setting based on prior information from the user. Details of S500 will be described later with reference to FIG. Next, in S501 (processing at the time of installation), each device of the image processing system 100 executes a process for confirming the operation of the system according to a command issued from the controller 140 based on an operation from the user. Details of S501 will be described later with reference to FIG. 7. Next, in S502 (pre-imaging processing), the virtual camera operation UI 142 outputs an image or sound before the start of imaging for a competition or the like. As a result, the user can confirm the sound picked up by the microphone 111 and the image captured by the camera 112 before the competition or the like.

In S503 (processing at the time of imaging), the control station 141 of the controller 140 causes each microphone 111 to collect sound and each camera 112 to perform imaging. Each camera adapter 120 connected to the camera 112 starts counting the time code (imaging time and frame number) which is the imaging timing information, starting from the imaging instruction from the control station 141. The image pickup in S503 includes sound collection using the microphone 111, but is not limited to this, and may be only image capture. When changing the setting made in S501 (YES in S504), the process proceeds to S506. When the imaging is finished (NO in S504, YES in S505), the process proceeds to S507. The determination in S504 and S505 is typically made based on the input from the user to the controller 140. However, it is not limited to this example.

In S506 (setting change processing), the controller 140 changes the setting made in S501. The content of the change is typically determined based on the user input acquired in S504. When it is necessary to stop the imaging when the setting is changed in S506, the imaging is stopped once, and the imaging is restarted after the setting is changed. If it is not necessary to stop the imaging, the setting is changed in parallel with the imaging.

In S507 (processing at the time of editing), the controller 140 edits the image captured by the plurality of cameras 112 and the sound picked up by the plurality of microphones 111. The editing is typically performed based on user operations entered via the virtual camera operation UI 142. The processes of S507 and S503 may be performed in parallel. For example, when a sports competition, a concert, or the like is distributed in real time (for example, an image of the competition is distributed during the competition), the imaging of S503 and the editing of S507 are performed at the same time. If the highlight image in the sports competition is delivered after the competition, editing is performed after the imaging is completed in S507.

Next, the details of the above-mentioned S500 (pre-installation process) will be described with reference to the flowchart of FIG. First, in S600, the control station 141 accepts input from the user regarding information (stadium information) regarding the facility to be imaged. The stadium information refers to the shape of the stadium, sound, lighting, power supply, transmission environment, three-dimensional model data of the stadium, and the like. That is, the stadium information includes the above-mentioned stadium shape data. In this embodiment, the case where the facility to be imaged is a stadium is described. This is based on the assumption that images of sports competitions held at the stadium will be generated. However, since some sports competitions are held indoors, the facilities to be imaged are not limited to stadiums. In addition, since the virtual viewpoint image of the concert in the concert hall may be generated and the image of the outdoor concert in the stadium may be generated, it is clarified that the event to be imaged is not limited to the competition. Keep it.

Next, in S601, the control station 141 receives an input from the user regarding the device information. The device information refers to information on imaging devices such as cameras, pan heads, lenses, and microphones, information devices such as LANs, PCs, servers, and cables, and relay vehicles. In particular, it accepts input of the number of front-end servers 131 and the MAC addresses of all front-end servers 131. In the present embodiment, the MAC address is used to specify the front-end server as the destination, but the present invention is not limited to this, and any address can be used as long as the bronte-end server can be uniquely specified in the network. Further, the control station 141 notifies all the camera adapters 120 of the MAC addresses of all the input front-end servers 131. In the configuration example of FIG. 1, the MAC addresses of the front-end servers 131a to 131b are notified to the camera adapters 120a to 120z. Further, the control station 141 inputs a scheduling method from the camera adapter 120 to the front-end server 131. The scheduling method is, for example, the correspondence between the frame number of the captured image and the front-end server 131 of the distribution destination. More specifically, for example, it means that the captured images are alternately transmitted to the front-end servers 131a and 131b in order from the frame number 1 in frame units.

Next, in S602, the control station 141 receives the input regarding the arrangement information of the camera, the pan head, and the microphone among the imaging devices to which the device information is input in S601. The placement information can be input using the above-mentioned three-dimensional model data of the stadium.

Next, in S603, the control station 141 accepts user input regarding the operation information of the image processing system 100. Operational information refers to the imaging target, frame rate, imaging time, camera work, gazing point, and the like. For example, when the image to be captured is an opening ceremony in which the foreground image of a player or the like is overwhelmingly larger than that of a match in the captured image, the image generation method can be changed to a method suitable for the situation. In addition, the gazing point may be changed and the restrictions on camera work may be changed according to the type of competition such as athletics or soccer using the field. A table of setting information composed of a combination of these operational information is managed, changed, and instructed by the control station 141. The workflow before system installation is completed by the above-mentioned S600 to S603.

Next, the details of S501 (processing at the time of installation) described above will be described with reference to the flowchart of FIG. First, in S700, the control station 141 accepts user input (equipment confirmation) regarding the presence or absence of excess or deficiency of installed equipment. The user can determine whether or not there is excess or deficiency of the installed equipment by comparing the equipment information input in S601 with the equipment to be installed and confirming whether or not there is excess or deficiency. Next, in S701, the control station 141 executes the installation confirmation process of the equipment determined to be insufficient in S700. That is, the user can install the missing equipment between S700 and S701, and the control station 141 confirms that the missing equipment has been installed by the user.

Next, in S702, the control station 141 activates the equipment installed in S701 and confirms the operation of the pre-adjustment system to see if it operates normally. In the process of S702, the user may confirm the system operation, and the user may input the confirmation result to the control station 141. Here, if an excess or deficiency of equipment or an error occurs in operation (NO in S702), the control station 141 notifies the user of the error in S703. The control station 141 is in a locked state in which it does not proceed to the next step until the error is cleared. If the error state is not detected, or if the error state is cleared (YES in S702), in S704, the control station 141 notifies the user normally and proceeds to the subsequent processing. This makes it possible to detect an error at an initial stage. After confirmation, the processing related to the camera 112 is executed in S705 to S707, and the processing related to the microphone 111 is executed in S708 to S710.

First, the processing related to the camera 112 will be described. In S705, the control station 141 adjusts the installed camera 112. The adjustment of the camera 112 of S705 includes, for example, angle-of-view matching and color matching, and is performed for all installed cameras 112. The adjustment of S705 may be performed based on a user operation, or may be realized by an automatic adjustment function. In the angle of view adjustment, zoom, pan, tilt, and focus adjustments are performed in parallel, and the adjustment results are stored in the control station 141. In color matching, IRIS, ISO / gain, white balance, sharpness, and shutter speed adjustments are performed in parallel, and the adjustment results are stored in the control station 141.

Next, in S706, the control station 141 adjusts so that all the installed cameras are synchronized. The synchronization adjustment in S706 may be realized by the automatic adjustment function, but may be performed based on the user operation. Further, in S707, the control station 141 calibrates when the camera is installed. More specifically, the control station 141 adjusts so that the coordinates of all the installed cameras match the world coordinates. Detailed calibration will be described later with reference to FIG. It should be noted that the control command of the camera 112 and the communication confirmation of the network route related to the synchronization with the time server are also carried out. Then, after the adjustment, the system operation normality confirmation process waits until the microphone adjustment proceeds (S711).

Next, the processing related to the microphone 111 will be described. First, in S708, the control station 141 adjusts the installed microphone 111. The adjustment of the microphone 111 of S708 includes, for example, a gain adjustment, and is performed for all the installed microphones. The adjustment of the microphone 111 in S708 may be realized by the automatic adjustment function, but may be performed based on the user operation.

Next, in S709, the control station 141 adjusts so that all the installed microphones are synchronized. Specifically, the synchronous clock is confirmed. The synchronization adjustment in S709 may be performed based on a user operation, or may be realized by an automatic adjustment function.

Next, in S710, the control station 141 adjusts the position of the microphone 111 installed in the field among the installed microphones 111. The adjustment of the position of the microphone 111 in S710 may be performed based on a user operation, or may be realized by an automatic adjustment function. It should be noted that the control command of the microphone 111 and the communication confirmation of the network route related to the synchronization with the time server are also carried out.

Next, in S711, the control station 141 performs an adjusted system operation check for the purpose of confirming whether the cameras 112a to 112z and the microphones 111a to 111z have been adjusted correctly. The process of S711 can be executed based on a user instruction. If it is confirmed that the system operation is normal after adjusting both the camera 112 and the microphone 111 (YES in S711), the control station 141 notifies the user of the normality in S713. On the other hand, when an error occurs (NO in S711), in S712, the control station 141 notifies the user of the error. In the error notification, the type and individual number of the camera 112 or microphone 111 in which the error occurred are notified. The control station 141 issues a readjustment instruction based on the type and individual number of the device in which the error has occurred.

Next, the installation processing (S501) and the pre-imaging processing (S502) will be described. The image processing system 100 can switch and control the state of performing calibration at the time of installation and the state of performing normal imaging by changing the operation mode. In some cases, it may be necessary to calibrate a specific camera during imaging. In this case, two types of operations, imaging and calibration, are compatible.

The calibration process at the time of installation will be described with reference to the sequence diagram shown in FIG. In FIG. 8, although the description of the notification of the completion of data reception and the completion of processing for the instruction given between the devices is omitted, it is assumed that some response is returned in response to the instruction.

First, when the installation of the camera 112 is completed, the user instructs the control station 141 to perform the installation calibration. Then, the control station 141 instructs the front-end server 131 and the camera adapter 120 to start calibration (S800).

Upon receiving the calibration start instruction, the front-end server 131 determines that the image data received thereafter is the calibration data, and changes the control mode so that the calibration unit 340 can process it (S802a). Although only one front-end server 131 is shown in FIG. 8, a plurality of front-end servers are used in the present embodiment. Therefore, the control station 141 gives an instruction to start calibration, an instruction to estimate camera parameters and an instruction to end calibration, which will be described later, to each of the plurality of front-end servers. Then, each of the plurality of front-end servers executes the process shown in FIG. Further, when the camera adapter 120 receives a calibration start instruction, it shifts to a control mode for handling an uncompressed frame image without performing image processing such as foreground background separation (S802b). Further, the camera adapter 120 instructs the camera 112 to change the camera mode (S801). Upon receiving this, the camera 112 changes the control mode to a mode for performing calibration (S802c). In this mode, for example, the frame rate of the camera 112 is set to 1 fps. Alternatively, the camera 112 may be set to transmit a still image instead of a moving image. Further, the frame rate may be controlled by the camera adapter 120 to set the mode in which the calibration image is transmitted.

The control station 141 instructs the camera adapter 120 to acquire the zoom value and the focus value of the camera (S803), and the camera adapter 120 transmits the zoom value and the focus value of the camera 112 to the control station 141 (S803). S804). In FIG. 8, only one camera adapter 120 and one camera 112 are described, but the control regarding the camera adapter 120 and the camera 112 is applied to all the camera adapters 120 and all the cameras 112 in the image processing system 100. Each is executed. Therefore, S803 and S804 are executed for the number of cameras, and when the processing of S803 and S804 for all the cameras 112 is completed, the control station 141 is in a state where the zoom value and the focus value for all the cameras can be received. .. Next, the control station 141 transmits the zoom value and the focus value for all the cameras received in S804 to the front-end server 131 (S805).

Next, the control station 141 notifies the front-end server 131 of the imaging pattern of the calibration imaging at the time of installation (S806). Here, the imaging pattern has an attribute of a pattern name (for example, patterns 1 to 10) for distinguishing images captured at different timings when a marker or the like, which is an image feature point, is moved in the ground to capture images multiple times. Is added. That is, the front-end server 131 determines that the image data for calibration received after S806 is the captured image in the imaging pattern received in S806.

The control station 141 instructs the camera adapter 120 to take a synchronized still image (S807), and the camera adapter 120 instructs the camera 112 to take a synchronized still image with all the cameras (S808). Then, the camera 112 transmits the captured image to the camera adapter 120 (S809). The control station 141 instructs the camera adapter 120 to transmit the image captured in S807 to all the front-end servers 131 (S810). Further, the camera adapter 120 transmits the image received in S809 to all the front-end servers 131 designated as transmission destinations (S811). When there are a plurality of gazing point groups, the calibration images of S806 to S811 may be captured for each gazing point group.

The calibration image transmitted in S811 is not subjected to image processing such as foreground background separation, and the captured image is transmitted as it is without being compressed. Therefore, when all cameras perform high-resolution imaging or when the number of cameras increases, there is a risk that all uncompressed images cannot be transmitted at the same time due to restrictions on the transmission band. As a result, the time required for calibration in the workflow may increase. In that case, in the image transmission instruction of S810, the transmission instruction of the uncompressed image according to the calibration pattern attribute is sequentially given to each of the camera adapters 120. Further, in such a case, since it is necessary to image more feature points according to the pattern attribute of the marker, image imaging for calibration using a plurality of markers is performed. In this case, from the viewpoint of load distribution, image imaging and uncompressed image transmission may be performed asynchronously. Further, the uncompressed image acquired by image imaging for calibration may be sequentially accumulated in the camera adapter 120 for each pattern attribute, and the uncompressed image may be transmitted in parallel in response to the image transmission instruction of S810. This has the effect of reducing workflow processing time and human error.

When the processing of S811 is completed in all the cameras 112, all the front-end servers 131 are in a state of being able to receive the captured images for all the cameras. As described above, when there are a plurality of imaging patterns, the processes of S806 to S811 are repeated for the number of patterns.

Next, when all the calibration imaging is completed, the control station 141 instructs the front-end server 131 to perform camera parameter estimation processing (S812). Upon receiving the camera parameter estimation processing instruction, the front-end server 131 performs camera parameter estimation processing using the zoom values and focus values for all cameras received in S805 and the captured images for all cameras received in S811. (S813). The details of the camera parameter estimation process in S813 will be described later. When there are a plurality of gazing points, the camera parameter estimation process of S813 is performed for each gazing point group. Then, the front-end server 131 transmits and stores the camera parameters for all the cameras derived as a result of the camera parameter estimation process in S813 to the database 132 (S814).

Further, the front-end server 131 also transmits camera parameters for all cameras to the control station 141 (S815). The control station 141 transmits the camera parameters corresponding to the camera 112 to the camera adapter 120 (S816). The camera adapter 120 receives and stores the camera parameters of the camera 112 connected to itself (S817).

Then, the control station 141 confirms the calibration result (S818). As a confirmation method, the derived numerical values of the camera parameters may be confirmed, the calculation process of the camera parameter estimation process of S814 may be confirmed, or an image is generated using the camera parameters. You may check the image. Then, the control station 141 instructs the camera adapter 120 and the front-end server 131 to end the calibration (S819).

Upon receiving the calibration end instruction, the front-end server 131 changes the control mode so as to determine that the image data received thereafter is not the calibration data, contrary to the calibration start process of S802a (S821a). When the camera adapter 120 receives the calibration end instruction, the camera adapter 120 shifts to the control mode for performing image processing such as foreground and background separation, contrary to the calibration start process executed in S802b (S821b). Further, the camera adapter 120 instructs the camera 112 to change the camera mode (S820). Upon receiving this, the camera 112 shifts to the normal imaging mode (S821c), contrary to the calibration start process executed in S802c.

By the above processing, as the calibration process at the time of installation, the camera parameters for all the cameras can be derived, and the derived camera parameters can be saved in the camera adapter 120 and the database 132.

In addition, the above-mentioned installation calibration process is performed after the camera is installed and before imaging, and if the camera is not moved, it is not necessary to process it again. However, when the camera is moved (for example, in the first half and the second half of the game) To change the viewpoint, etc.), the same process is performed again.

Further, when the camera 112 moves beyond a predetermined threshold value due to an accident such as a ball hitting during imaging, the camera 112 is changed from the imaging state to the calibration start state and the above-mentioned installation calibration is performed. Is also good. In that case, it is necessary to put the entire system into the calibration mode by maintaining the normal imaging state as the system and notifying the front-end server 131 that only the camera 112 is transmitting the calibration image. The continuity of imaging can be achieved without any problem. Furthermore, in the transmission of this system in the daisy chain, if an uncompressed image for calibration is sent to the transmission band of image data in normal imaging, the transmission band limit may be exceeded. In this case, the transmission priority of the uncompressed image is lowered, or the uncompressed image is divided and transmitted. Furthermore, when the connection between the camera adapters 120 is 10 GbE or the like, the bandwidth can be secured by transmitting an uncompressed image in the opposite direction to the image data transmission of normal imaging by using the full-duplex feature. ..

Further, when it is desired to change the gazing point of one of the plurality of gazing points, the above-described installation calibration process may be performed again only for the camera 112 of one gazing point group. In that case, during the calibration process, it is not possible to perform normal image imaging and virtual viewpoint image generation for the camera 112 of the target gazing point group. Therefore, the control station 141 is notified that the calibration process is in progress, and the control station 141 requests a process such as restricting the viewpoint operation on the virtual camera operation UI 142. The front-end server 131 controls the camera parameter estimation process so as not to affect the virtual viewpoint image generation process.

Next, imaging by the camera 112, sound collection by the microphone 111, and processing of accumulating the imaged or collected data in the database 132 via the camera adapter 120 and the front-end server 131 will be described.

The imaging start processing sequence of the camera 112 will be described with reference to FIGS. 9A and 9B. Although FIGS. 9A and 9B show processing sequences having different contents, similar results can be obtained according to any of the sequences. The camera adapter 120 selects whether to perform the process shown in FIG. 9A or the process shown in FIG. 9B according to the specifications of the camera 112. For example, the camera adapter 120 performs the process of FIG. 9A when the connected camera 112 can add meta information including a time code (Time Code) to the captured image, and the process of FIG. 9A otherwise. The process of 9B is executed.

First, FIG. 9A will be described. The time server 134 synchronizes the time with, for example, GPS900, and sets the time managed in the time server 134 (S901). The time is not limited to the method using GPS900, and the time may be set by another method such as NTP (Network Time Protocol). Next, the camera adapter 120 communicates with the time server 134 using PTP (Precision Time Protocol), corrects the time managed in the camera adapter 120, and synchronizes the time with the time server 134 (S902).

The camera adapter 120 starts to provide the camera 112 with a synchronized imaging signal such as a Genlock signal and a ternary synchronization signal and a time code signal in synchronization with the imaging frame (S903). The information provided is not limited to the time code, and other information may be used as long as it is an identifier that can identify the imaging frame. Next, the camera adapter 120 gives an imaging start instruction to the camera 112 (S904). When the camera 112 receives an imaging start instruction, it performs imaging in synchronization with the Genlock signal (S905).

Next, the camera 112 includes a time code signal in the captured image and transmits it as image data (captured image) to the camera adapter 120 (S906). Imaging synchronized with the Genlock signal is performed until the camera 112 stops imaging. The camera adapter 120 performs PTP time correction processing with the time server 134 during imaging to correct the generation timing of the Genlock signal (S907). When the required correction amount becomes large, the correction according to the preset change amount may be applied.

As described above, it is possible to realize synchronous imaging of a plurality of cameras 112 connected to a plurality of camera adapters 120 in the system.

Next, FIG. 9B will be described. First, as in the case of FIG. 9A, the time synchronization process is performed between the camera adapter 120, the time server 134, and the GPS 900 (S901, S902). Next, the camera adapter 120 gives an imaging start instruction (S953). The imaging start instruction includes information for specifying the imaging period and the number of frames. The camera 112 performs imaging according to the imaging start instruction (S954), and transmits the captured image data (captured image) to the camera adapter 120 (S955). The camera adapter 120 that has received the image data adds a time code to the meta information of the image data (S956).

The camera adapter 120 performs PTP time correction processing with the time server 134 during imaging, and corrects the imaging timing for the camera 112. When the required correction amount becomes large, the correction according to the preset change amount may be applied. For example, the imaging start instruction is repeatedly performed at a short timing such as every frame, and the above-mentioned processes S953 to S956 are repeated (S957 to S960).

Although the imaging start processing sequence of the camera 112 has been described in FIGS. 9A and 9B, the microphone 111 also performs the same processing as the synchronous imaging of the camera 112 to perform synchronous sound collection.

Next, a processing sequence in which a plurality of camera adapters 120 (120a, 120b, 120c, and 120d) in the present embodiment work together to generate three-dimensional model information will be described with reference to FIG. The order of processing is not limited to that shown in the figure.

The image processing system 100 of the present embodiment includes 26 cameras 112 and a camera adapter 120, but here, two cameras 112b and 112c, and four camera adapters 120a, 120b, 120c, and The explanation will be given with a focus on 120d. The four camera adapters are connected by a daisy chain, and the camera adapters 120a, 120b, 120c, and 120d are connected in this order from upstream to downstream. Further, the camera 112b is connected to the camera adapter 120b, and the camera 112c is connected to the camera adapter 120c, respectively. The cameras 112a and 112d connected to the camera adapter 120a and the camera adapter 120d, the microphone 111 connected to each camera adapter 120, the pan head 113, and the external sensor 114 will be omitted. Further, it is assumed that the camera adapters 120a to 120d have completed time synchronization with the time server 134 and are in the imaging state.

The camera 112b and the camera 112c transmit the captured image (1) and the captured image (2) to the camera adapters 120b and 120c connected downstream thereof (S1001, S1002). In the camera adapters 120b and 120c, the calibration control units 233 perform calibration processing on the received captured images (1) and captured images (2) (S1003, S1004). The calibration process is, for example, color correction or blur correction. Although the calibration process is performed in this embodiment, it is not always necessary to perform the calibration process. Next, the foreground background separation unit 231 of the camera adapters 120b and 120c performs the foreground background separation processing on the calibrated captured image (1) and the captured image (2) (S1005, S1006).

Next, in the camera adapters 120b and 120c, the data compression / decompression unit 221 compresses the foreground image and the background image separated by the foreground / background separation unit 231 (S1007, S1008). The compression ratio may be changed according to the importance of the separated foreground image and background image. Further, in some cases, compression may not be performed. For example, the camera adapter 120 compresses at least the background image of the foreground image and the background image so that the compression ratio of the foreground image is lower than that of the background image, and outputs the compressed image to the downstream camera adapter 120. When both the foreground image and the background image are compressed, lossless compression is performed on the foreground image including an important imaging target, and lossy compression is performed on the background image not including the imaging target. As a result, the amount of data transmitted to the next camera adapter 120c or camera adapter 120d after that can be efficiently reduced. For example, when the field of a stadium where soccer, rugby, baseball, etc. are held is imaged, most of the image is composed of a background image, and the area of the foreground image of a player or the like is small. It is clearly stated here that a large reduction can be made.

Further, the camera adapter 120b or the camera adapter 120c may change the frame rate of the image output to the camera adapter 120c or the camera adapter 120d connected downstream thereof, depending on the importance. For example, the foreground image including the important imaging target may be output at a high frame rate, and the background image not including the imaging target may be output at a low frame rate. As a result, the amount of data transmitted to the next camera adapter 120c or camera adapter 120d can be further reduced. Further, the compression rate and the transmission frame rate may be changed for each camera adapter 120 according to the installation location of the camera 112, the imaging location, and / or the performance of the camera 112. Further, since the three-dimensional structure of the spectator seats and the like of the stadium can be confirmed in advance by using drawings, the camera adapter 120 may transmit an image excluding the spectator seat portion from the background image. As a result, at the time of rendering described later, image rendering focused on the player during the match is performed by using the stadium three-dimensional structure generated in advance, and the amount of data transmitted and stored in the entire system can be reduced. The effect of being able to do it is born.

Next, the camera adapter 120 transfers the compressed foreground image and background image to the adjacent camera adapter 120 downstream (S1010, S1011, S1012). In the present embodiment, the foreground image and the background image are transferred at the same time, but each may be transferred individually.

Next, the camera adapter 120b creates three-dimensional model information using the foreground image received from the upstream camera adapter 120a and the foreground image separated by the foreground background separation process S1005 (S1013). Similarly, the camera adapter 120c also creates three-dimensional model information (S1014).

Next, the camera adapter 120b transfers the foreground image and the background image received from the upstream camera adapter 120a to the downstream camera adapter 120c (S1015). The camera adapter 120c also transfers the foreground image and the background image to the camera adapter 120d (S1016). In the present embodiment, the foreground image and the background image are transferred at the same time, but each may be transferred individually. Further, the camera adapter 120c is created by the camera adapter 120a and transfers the foreground image and the background image received from the camera adapter 120b to the camera adapter 120d (S1017).

Next, each of the camera adapters 120a to 120c transfers the created three-dimensional model information to the camera adapters 120b to 120d connected downstream (S1018, S1019, S1020). Further, the camera adapters 120b and 120c transfer the sequentially received three-dimensional model information to the next camera adapters 120c and 120d (S1021, S1022). Further, the camera adapter 120c transfers the three-dimensional model information created by the camera adapter 120a and received from the camera adapter 120b to the camera adapter 120d (S1023).

In this way, finally, the foreground image, the background image, and the three-dimensional model information created by the camera adapters 120a to 120d are sequentially transmitted between the network-connected camera adapters 120 and transmitted to the front-end server 131.

In this sequence diagram, the calibration process, the foreground background separation process, the compression process, and the three-dimensional model information creation process of the camera adapter 120a and the camera adapter 120d are omitted. However, in reality, the camera adapter 120a and the camera adapter 120d also perform the same processing as the camera adapter 120b and the camera adapter 120c to create a foreground image, a background image, and three-dimensional model information. Further, although the data transfer sequence between the four camera adapters 120 has been described here, the same processing is performed even if the number of camera adapters 120 increases.

As described above, among the plurality of camera adapters 120, the camera adapters 120 other than the final stage camera adapters 120 in a predetermined order extract a predetermined region from the image captured by the corresponding camera 112. Then, the image data based on the extraction result is output to the next camera adapter 120 in the predetermined order described above. On the other hand, the camera adapter 120 in the final stage in the predetermined order described above outputs image data based on the extraction result to the front-end server 131. That is, the plurality of camera adapters 120 are connected by a daisy chain, and the image data based on the result of each camera adapter 120 extracting a predetermined area from the captured image is obtained by the front-end server 131 by the predetermined final stage camera adapter 120. Is entered in. By using such a data transmission method, it is possible to suppress fluctuations in the processing load in the front-end server 131 and the transmission load in the network when the number of sensor systems 110 in the image processing system 100 fluctuates.

Further, the image data output by the camera adapter 120 is generated by using the image data based on the above extraction result and the image data based on the extraction result of the predetermined area by the previous camera adapter 120 in a predetermined order. It may be data. For example, each camera adapter 120 can reduce the amount of data transmitted in the system by outputting image data based on the difference between the extraction result by itself and the extraction result by the previous camera adapter 120. In the above order, the camera adapter 120 in the final stage acquires the extracted image data based on the image data of the predetermined region extracted by the other camera adapter 120 from the images captured by the other camera 112 from the other camera adapter 120. .. Then, the image data corresponding to the extraction result of the predetermined area extracted by itself and the extracted image data acquired from the other camera adapter 120 is output to the image computing server 130 for generating the virtual viewpoint image. ..

Further, the camera adapter 120 divides the image captured by the camera 112 into a foreground portion and a background portion, and changes the compression rate and the frame rate to be transmitted according to the importance of each. As a result, the amount of transmission can be reduced as compared with the case where all the data captured by the camera 112 is transmitted to the front-end server 131. Further, each camera adapter 120 sequentially creates the three-dimensional model information necessary for generating the three-dimensional model. As a result, it is possible to reduce the processing load of the server compared to the case where all the data is collected in the front-end server 131 and all the 3D model generation processing is performed by the server, and the 3D model generation can be performed in more real time. Can be made possible.

Next, the process of distributing the image data output from the camera adapter 120 to the plurality of front-end servers 131 will be described with reference to FIGS. 10 to 12. In the present embodiment, the case where the number of camera adapters is three and the number of front-end servers is two will be described, but the number of camera adapters and front-end servers is not limited to this.

FIG. 11 shows an example of the data format output by the camera adapter 120. Reference numeral 1101 is a destination MAC address, which stores the MAC address of the front-end server 131a or 131b to which the image data is transmitted. These MAC addresses are the MAC addresses notified from the control station 141 to the camera adapter 120 in S601 of FIG. In this embodiment, the MAC addresses of all the front-end servers 131a to 131b are notified to all the camera adapters 120a to 120z. 1102 is the source MAC address, and stores the MAC address of the camera adapter 120, which is the source of the image data. 1103 is an IP header, and stores the IP addresses of the source and the destination.

Reference numeral 1104 is a frame number of the image data, and the frame number added for each frame is stored based on the start point of the start of moving image imaging instructed in S714 of FIG. Reference numeral 1105 is a camera number, which is a number assigned to each of the cameras 112a to 112z. The camera number may be a number assigned in advance such as the manufacturing serial number of each camera 112, or a number input from the control station 141 in S601 of FIG. Further, when the camera 112 and the camera adapter 120 are connected one-to-one, the number of the camera adapter 120 may be used instead of the number of the camera 112. Reference numeral 1106 is a data entity of image data (foreground image, background image, and three-dimensional model information). Although only characteristic data is described and described in FIG. 11, information other than the above may be included.

Next, the operation of the camera adapter 120 will be described with reference to the flowchart of FIG. 12 and the sequence diagram of FIG. In particular, in the sequence diagram of FIG. 10, the camera adapter 120b will be focused on. In the description of FIG. 12, the data transmitted and received by the camera adapter 120 is a general term for the foreground image, the background image, and the three-dimensional model information.

In S1201, the camera adapter 120 shifts to the promiscuous mode. This transition is typically performed starting from a user operation, but the camera adapter 120 may be automatically set to the promiscuous mode when the power is turned on. In S1202, the camera adapter 120 (external device control unit 240) inputs an captured image from the connected camera 112. This is a process corresponding to S1001 in FIG. In S1203, the camera adapter 120 (network adapter 210) receives image data from the adjacent camera adapter 120. This is a process corresponding to S1010 and S1018 in FIG. In S1204, the camera adapter 120 (image processing unit 230) performs image processing on the received image data. This is a process corresponding to S1003, S1005, S1007, and S1013 in FIG. The processes of S1202 to S1204 are processes that are appropriately executed while the processes shown in FIG. 12 are repeated. That is, the processing of S1202 and S1204 is a processing that is appropriately executed when synchronous imaging by the camera 112 is executed and the captured image is received, and the processing of S1203 is appropriately executed when data is transferred from the upstream camera adapter. ..

In S1205, the camera adapter 120 (transmission unit 220) determines whether or not the image data to be transmitted is the image data processed in S1204. When transmitting the image data processed in S1204 (YES in S1204), the process proceeds to S1206, and in other cases (NO in S1204), the process proceeds to S1209. In S1209, the camera adapter 120 (transmission unit 220) transmits the image data received in S1203. This is a process corresponding to S1015 and S1021 in FIG.

In S1206, the camera adapter 120 (transmission unit 220) determines the front-end server 131 for distributing data based on the scheduling method input from the control station 141 in S601 of FIG. Since the front-end server 131 synthesizes the images captured by the plurality of cameras 112 to generate one image, it is necessary to transmit the images of the same frame to the same server and perform the compositing process. Therefore, in the camera adapter 120, for example, based on the imaging timing information being the first imaging timing information, the imaging timing information is the second imaging to the first front-end server among the plurality of front-end servers 131. Based on the timing information, the image data is controlled to be transmitted to the second front-end server among the plurality of front-end servers 131. As an example of the scheduling method, there is a method of determining the front-end server 131 to which the image data is distributed according to the remainder obtained by dividing the frame number of the image data by the number of front-end servers. For example, when there are two front-end servers 131a and 131b, the odd-numbered frames are the front-end server 131a, the even-numbered frames are the front-end server 131b, and the distribution destination front-end server 131 is determined. As another example of the scheduling method, the control station 141 creates a schedule table that defines the correspondence between the imaging timing information (for example, the frame number) and the front-end server 131 of the transmission destination, and the camera adapter 120 follows the schedule table. Image data may be sorted. In the schedule table, for example, the correspondence between the first imaging timing information (odd frame number) and the first front-end server (131a), the second imaging timing information (even frame number) and the second front end The correspondence of the server (131b) and the like are described.

In S1207, the camera adapter 120 (transmission unit 220) generates data in which the MAC address of the distribution destination front-end server 131a or 131b determined in S1206 is set as the destination MAC address. In S1208, the camera adapter 120 (transmission unit 220) transmits the data generated in S1207. This is a process corresponding to S1011 and S1019 in FIG. The processes of S120 to S1209 are processes that are appropriately executed according to the generation of data to be transmitted while the processes shown in FIG. 12 are repeated.

Next, the data flow of the camera adapter 120 and the front-end server 131 according to the operation of the flowchart of FIG. 12 will be described with reference to FIGS. 13A and 13B.

FIG. 13A shows how each camera 112 captures an image having a frame number “m” and transmits image data from each camera adapter 120 to the next camera adapter.

1301, 1302, 1303, 1304, and 1305 represent MAC addresses assigned to the camera adapters 120a to 120c and the front-end servers 131a to 131b, respectively.

Reference numerals 1310, 1320, and 1330 represent data to be transmitted by the camera adapters 120a, 120b, and 120c, respectively, and the data format conforms to the format described with reference to FIG. Further, the data 1310, 1320, 1330 correspond to the image data transmitted in S1208 of the flowchart of FIG.

Further, 1311, 1321, 1331 are MAC addresses of the front-end server 131 that is the destination of the image data. In FIG. 13A, the MAC address “00:00:00:20:01” of the front-end server 131a is set to transmit the image data of the frame number “m” to the front-end server 131a based on the processes of S1206 to S1207. It is set. As a result, all the image data of the frame number "m" is transmitted to the front-end server 131a. Reference numerals 1312, 1322, and 1332 are MAC addresses of the camera adapter 120 that is the source of image data, and the MAC address of the camera adapter 120 that has captured and processed the image is set.

FIG. 13B shows how the image data of the frame number “m + 1” captured by each camera 112 and the image data of the frame number “m” received from the adjacent camera adapter 120 are transmitted to the next camera adapter. ing. The data 1310 and 1320 shown in FIG. 13A are transferred to the downstream camera adapters, respectively. 1340, 1350, and 1360 are the data of the frame number “m + 1” transmitted by the camera adapters 120a, 120b, and 120c, respectively, after performing image processing, and correspond to the image data transmitted in S1208 of the flowchart of FIG. 1341, 1351, and 1361 are MAC addresses of the front-end server 131 that is the destination of the image data. Based on the processing of S1206 to S1207, the MAC address "00:00:00:20:02" of the front-end server 131b is set in order to transmit the image data of the frame number "m + 1" to the front-end server 131b. ..

Reference numerals 1360 and 1370 are image data received from the adjacent camera adapter 120, and the contents are transferred as they are without being rewritten. This corresponds to the data transmitted in S1209 of the flowchart of FIG. The data 1350 and 1360 in FIG. 13B correspond to the data 1310 and 1320 in FIG. 13A, respectively, and the data contents are the same.

As described above, all the image data of the frame number "m" is transmitted to the front-end server 131a, and all the image data of the frame number "m + 1" is transmitted to the front-end server 131b. After that, by repeating the same processing, the image data can be alternately transmitted to the front-end servers 131a and 131b for each frame, and the processing can be distributed by the plurality of front-end servers 131.

Next, the operation of the front-end server 131 in the imaging process (S503) will be described with reference to the flowchart of FIG.

The control unit 310 receives an instruction to switch to the imaging mode from the control station 141 and switches to the imaging mode (S1400). When the imaging is started, the data input control unit 320 starts receiving the imaging data from the camera adapter 120 (S1401). The data synchronization unit 330 buffers the imaging data until all the imaging data required for file creation are prepared (S1402). Although not specified in the flowchart, in S1402, it is determined whether or not the time information given to the imaging data matches, and whether or not a predetermined number of cameras are satisfied.

Further, depending on the state of the camera 112, image data may not be sent due to calibration or error processing. In this case, it is notified in the data transfer to the database 132 in the subsequent stage (S1408) that the image of the predetermined camera number is missing. Here, there is a method of waiting for the arrival of the imaging data for a predetermined time in order to determine the satisfaction of the predetermined number of cameras. However, in the present embodiment, in order to suppress the delay of a series of processing of the system, when each camera adapter 120 transmits data by the daisy chain, information indicating the presence or absence of image data corresponding to each camera number is added. To do. As a result, the control unit 310 of the front-end server 131 can immediately determine the presence or absence of image data. It is clearly stated here that this has the effect of eliminating the need to set the arrival waiting time of the imaging data.

After the data required for file creation is buffered by the data synchronization unit 330, the image processing unit 350 develops the RAW image data, corrects the lens distortion, and captures the foreground image and the background image between the images captured by the cameras. Various conversion processes such as matching color and brightness values are performed (S1403). When the data buffered by the data synchronization unit 330 includes the background image (YES in S1404), the image combining unit 370 performs a process of combining the background images (S1405). The image combining unit 370 acquires the background image processed by the image processing unit 350 in S1403, and joins the background images according to the coordinates of the stadium shape data saved by the CAD data storage unit 335 to obtain the combined background image. .. The combined background image is sent to the imaging data file generation unit 380.

On the other hand, if the buffered data does not include the background image (NO in S1404), the combination of the background images (S1405) is skipped.

Next, the three-dimensional model joining unit 360 performs a process of joining the three-dimensional models. The three-dimensional model combining unit 360 that has acquired the three-dimensional model from the data synchronization unit 330 generates and combines the three-dimensional model of the foreground image using the three-dimensional model data and the camera parameters (S1406). The imaging data file generation unit 380 that has received the imaging data created by the processes up to S1406 shapes the imaging data according to the file format, packs it, and sends it to the DB access control unit 390 as an imaging data file (S1407). .. The DB access control unit 390 transmits the image pickup data file received from the image pickup data file generation unit 380 in S1407 to the database 132 (S1408).

Subsequently, the hardware configuration of each device constituting the present embodiment will be described in more detail. As described above, in the present embodiment, an example in which the camera adapter 120 implements hardware such as FPGA and / or ASIC and executes each of the above-described processes by these hardware has been mainly described. The same applies to the various devices in the sensor system 110, the front-end server 131, the database 132, the back-end server 133, and the controller 140. However, at least one of the above devices may execute the processing of the present embodiment by software processing using, for example, a CPU, GPU, DSP, or the like.

FIG. 15 is a block diagram showing a hardware configuration example of the camera adapter 120 for realizing the functional configuration shown in FIG. 2 by software processing. Devices such as the front-end server 131, the database 132, the back-end server 133, the control station 141, the virtual camera operation UI 142, and the end-user terminal 150 may also have the hardware configuration shown in FIG. The camera adapter 120 includes a CPU 1501, a ROM 1502, a RAM 1503, an auxiliary storage device 1504, a display unit 1505, an operation unit 1506, a communication unit 1507, and a bus 1508.

The CPU 1501 controls the entire camera adapter 120 by using computer programs and data stored in the ROM 1502 and the RAM 1503. The ROM 1502 stores programs and parameters that do not need to be changed. The RAM 1503 temporarily stores programs and data supplied from the auxiliary storage device 1504, data supplied from the outside via the communication unit 1507, and the like. The auxiliary storage device 1504 is composed of, for example, a hard disk drive or the like, and stores content data such as still images and moving images.

The display unit 1505 is composed of, for example, a liquid crystal display or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the camera adapter 120. The operation unit 1506 is composed of, for example, a keyboard, a mouse, or the like, and inputs various instructions to the CPU 1501 in response to an operation by the user. The communication unit 1507 communicates with an external device such as the camera 112 or the front-end server 131. For example, when the camera adapter 120 is connected to an external device by wire, a LAN cable or the like is connected to the communication unit 1507. When the camera adapter 120 has a function of wirelessly communicating with an external device, the communication unit 1507 includes an antenna. The bus 1508 connects each part of the camera adapter 120 to transmit information.

For example, a part of the processing of the camera adapter 120 may be performed by the FPGA, and another part of the processing may be realized by software processing using the CPU. Further, in the present embodiment, the display unit 1505 and the operation unit 1506 are present inside the camera adapter 120, but the camera adapter 120 does not have to include at least one of the display unit 1505 and the operation unit 1506. Further, at least one of the display unit 1505 and the operation unit 1506 exists as another device outside the camera adapter 120, and the CPU 1501 controls the display control unit 1505 and the operation control unit 1506. It may operate as a unit.

Further, the above-described embodiment has been described focusing on an example in which the image processing system 100 is installed in a facility such as a stadium or a concert hall. Other examples of facilities include, for example, amusement parks, parks, racetracks, keirin racetracks, casinos, pools, skating rinks, ski resorts, live houses, and the like. In addition, the events held at various facilities may be held indoors or outdoors. The facilities in this embodiment also include facilities that are temporarily constructed (for a limited time).

<Second Embodiment>
In the first embodiment, a method in which each camera adapter 120 operates in the promiscuous mode and the MAC address of the destination front-end server 131 is directly specified to transmit image data has been described. Therefore, in the first embodiment, the connection order of the plurality of camera adapters defined by the scheduling method is realized by the connection order of the daisy chain. In the second embodiment, a configuration that does not use the promiscuous mode will be described. In this case, in the second embodiment, the order from upstream to downstream between the camera adapters defined by the scheduling method is realized by the address set in the "destination MAC" by each adapter. Therefore, the daisy chain connection form is not essential. Further, in the second embodiment, only the most downstream, that is, the final stage camera adapter 120 specifies the MAC address of the front-end server 131. In the second embodiment, the plurality of camera adapters 120 may not be connected in a daisy chain connection form.

In the second embodiment, in S601 of FIG. 6, the MAC addresses of the front-end servers 131a and 131b are notified only to the final stage camera adapter 120z that transmits data to the front-end server 131. Camera adapters other than the camera adapter in the final stage are notified of addresses to which data is transmitted in the order specified by the scheduling method.

FIG. 16 is a flowchart showing the operation of the camera adapter 120 in the second embodiment. The same reference numerals are given to the same processes as in the first embodiment (FIG. 12). Further, similarly to FIG. 12, in the description of FIG. 16, the data transmitted and received by the camera adapter 120 is described as image data, but the foreground image, the background image, and the three-dimensional model information are generically referred to.

After executing S1202 to S1204, in S1601, the camera adapter 120 determines whether or not it is the final stage camera adapter. The information on whether or not the camera adapter is the final stage is set from the control station 141 to the camera adapter 120 at the time of inputting the arrangement information in S602 of FIG. For example, in the system configuration shown in FIG. 1, when the data of the camera adapter is transmitted in the order of camera adapter 120a → camera adapter 120b → ... → camera adapter 120z, the setting is that the camera adapter 120z is the final stage camera adapter. Is done.

When it is determined that the camera adapter is not the final stage camera adapter (NO in S1601), in S1602, the camera adapter 120 sets the downstream camera adapter according to the scheduling method to the destination MAC address of the image data. For example, the camera adapter 120a sets the MAC address of the camera adapter 120b as the destination of the image data. Then, in S1603, the camera adapter 120 transmits image data (S1603). The format of the image data transmitted here is the same as that of the first embodiment (FIG. 11).

On the other hand, if it is determined that the camera adapter is the final stage camera adapter (YES in S1601), the image data is transmitted to the front-end server determined as the destination according to the set scheduling method (S1604 to SS1606). The processing of S1604 to S1605 is the same as that of S1206 to S1207 of FIG.

Next, the data flow of the camera adapter 120 and the front-end server 131 according to the operation of the flowchart of FIG. 16 will be described with reference to FIG. FIG. 17 shows how the image data of the frame number “n + 1” captured by each camera 112 and the image data of the frame number “n” received from the adjacent camera adapter 120 are transmitted to the next camera adapter. ing. The same configuration and the same data as those in the first embodiment (FIGS. 13A and 13B) are given the same reference numerals.

1701 is the MAC address of the camera adapter 120z. 1710, 1720, and 1730 are data transmitted by the camera adapter 120 other than the camera adapter 120z to the downstream camera adapter 120. These correspond to the data transmitted in S1603 of the flowchart of FIG. The MAC address of the next camera adapter 120b is stored in the destination MAC address 1711 of the data 1710. Similarly, the MAC addresses of the next camera adapter 120z are stored in the destination MAC addresses 1721 and 1731 of the data 1720 and 1730, respectively.

The data 1740 and 1750 are the data transmitted by the camera adapter 120z in the final stage to the front-end server 131, and correspond to the data transmitted in S1606 of the flowchart of FIG. The camera adapter 120z performs distribution processing to the front-end server 131, and distributes the data 1740 of the frame number “n” to the front-end server 131a and the data 1750 of the frame number “n + 1” to the front-end server 131a. Therefore, the MAC address of the front-end server 131a is stored in the destination MAC address 1741 of the data 1740, and the MAC address of the front-end server 131b is stored in the destination MAC address 1751 of the data 1750.

As described above, the camera adapter 120z in the final stage can specify the MAC address of the front-end server 131 in the order of the camera adapters 120 defined by the scheduling method without using the promiscuous mode, and data can be distributed.

As described above, according to the above-described embodiment, the virtual viewpoint image can be easily generated regardless of the scale of the devices constituting the system such as the number of cameras 112, and the output resolution and output frame rate of the captured image. ..

As described above, according to each of the above embodiments, in an image processing system including a plurality of image pickup devices (camera 112) and a plurality of image processing devices (front end server 131), captured images captured at the same timing are captured. It is possible to distribute to one image processing device. As a result, there is an effect that the efficiency of frame-by-frame processing in the image processing system 100 is improved.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100 ... image processing system, 110 ... sensor system, 111 ... microphone, 112 ... camera, 113 ... pan head, 120 ... camera adapter, 180 ... switching hub, 150. .. End user terminal, 131 ... Front end server, 132 ... Database, 133 ... Back end server, 134 ... Time server, 141 ... Control station, 142 ... Virtual camera operation UI

Claims (16)

A plurality of control devices corresponding to a plurality of image pickup devices and acquiring a plurality of images acquired by being imaged by the plurality of image pickup devices, and a plurality of control devices.
A plurality of image processing devices for acquiring the plurality of images transmitted from the plurality of control devices, and a plurality of image processing devices.
Have,
The plurality of control devices transmit the image to the first image processing device included in the plurality of image processing devices based on the imaging timing information given to the image being the first image processing timing information. Then, based on the imaging timing information given to the image being the second imaging timing information, the image is transmitted to the second image processing device included in the plurality of image processing devices. Image processing system. The claim is characterized in that an image processing device to which an image to which the image pickup timing information is attached is transmitted is determined based on the image pickup timing information given to the image and the number of the plurality of image processing devices. The image processing system according to 1. The plurality of control devices correspond to the image pickup timing information of the plurality of image processing devices with the image to which the image pickup timing information is given based on the image pickup timing information and the schedule table given to the image. Transmit to the image processing device
The claim is characterized in that the schedule table includes a correspondence between the first imaging timing information and the first image processing apparatus, and a correspondence between the second imaging timing information and the second image processing apparatus. The image processing system according to 1. The image processing system according to any one of claims 1 to 3, wherein the image pickup timing information is a time code or a frame number assigned to the image. The image processing system according to any one of claims 1 to 4, wherein the control device sets the address of the image processing device as the transmission destination to the destination of the image acquired from the corresponding image pickup device. The image processing system according to claim 5, wherein the plurality of control devices are connected by a daisy chain. The image processing according to claim 6, wherein the control device transmits the image transmitted from the upstream control device to the downstream control device without changing the destination set in the daisy chain. system. The control device transmits images from the upstream control device to the downstream control device in a predetermined order.
The image pickup timing information added to the images of the image received from the upstream control device and the image acquired from the corresponding image pickup device by the final stage control device in the predetermined order is the first image pickup timing information. 1 to 4, wherein the information is transmitted to the first image processing apparatus based on the above, and is transmitted to the second image processing apparatus based on the second imaging timing information. The image processing system according to any one of the above items. An acquisition means for acquiring an image captured based on the image taken by the image pickup device, and
Based on the imaging timing information of the captured image being the first imaging timing information, the image is transmitted to the first image processing apparatus included in the plurality of image processing devices, and the imaging timing information is the second imaging. A determination means for determining the destination of the captured image so as to transmit the image to the second image processing device included in the plurality of image processing devices based on the timing information.
A control device including an output means for setting and outputting a destination determined by the determination means to data including the captured image. The control device according to claim 9, wherein the acquisition means acquires the captured image to which the imaging timing information is added from the imaging device by notifying the imaging device of a time code. The control device according to claim 9, further comprising an imparting means for imparting the imaging timing information to the captured image acquired by the acquiring means. Connected to other controllers in a daisy chain,
The control device according to any one of claims 9 to 11, wherein the data including the captured image transmitted from the control device upstream of the daisy chain is directly transmitted to the control device downstream of the daisy chain. A first connecting means for connecting to another control device and a second connecting means for connecting to the plurality of image processing devices are provided.
Further, in the determination means, the destination of the data received via the first connection means is the image pickup timing information given to the image captured image included in the received data as the first image capture timing information. It is characterized in that an image is transmitted to the first image processing device based on the above, and it is determined to be transmitted to the second image processing device based on the second imaging timing information. The control device according to any one of claims 9 to 11. An acquisition step of acquiring a plurality of images acquired by being imaged by the plurality of imaging devices corresponding to a plurality of imaging devices, and
Based on the imaging timing information given to the image acquired in the acquisition step being the first imaging timing information, the image is transmitted to the first image processing device included in the plurality of image processing devices. The first transmission process and
Based on the image pickup timing information given to the image acquired in the acquisition step being the second image pickup timing information, the image is transmitted to the second image processing apparatus included in the plurality of image processing apparatus. An image transmission method comprising a second transmission step. It is a control method of the control device.
The acquisition process to acquire the captured image based on the imaging of the imaging device, and
Based on the imaging timing information of the captured image being the first imaging timing information, the image is transmitted to the first image processing apparatus included in the plurality of image processing devices, and the imaging timing information is the second imaging. A determination step of determining the destination of the captured image so as to transmit the image to the second image processing device included in the plurality of image processing devices based on the timing information.
A control method for a control device, comprising: an output step of setting a destination determined by the determination step on data including the captured image and outputting the data. A program for causing a computer to function as each means of the control device according to any one of claims 9 to 13.

Images