JP6632703B2 - CONTROL DEVICE, CONTROL METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a system for generating a virtual viewpoint image.

  In recent years, attention has been paid to a technique of installing a plurality of cameras at different positions, performing synchronous shooting from multiple viewpoints, and generating virtual viewpoint content using the multiple viewpoint images obtained by the shooting. According to the technology for generating virtual viewpoint content from a plurality of viewpoint images as described above, for example, a highlight scene of soccer or basketball can be viewed from various angles. High realism can be given.

  On the other hand, generation and browsing of virtual viewpoint content based on a plurality of viewpoint images are performed by collecting images taken by a plurality of cameras into an image processing unit such as a server, and performing processing such as three-dimensional model generation and rendering in the image processing unit. And transmitting it to the user terminal.

  In Patent Document 1, a plurality of cameras are connected by an optical fiber via a control unit that is paired with each of the cameras, image frames of each camera are stored in the control unit, and continuous images are stored using the stored image frames. It describes that an image output expressing motion is performed.

U.S. Pat. No. 7,106,361

  However, in the related art, it is conceivable that an operator who sets a system for generating a virtual viewpoint image may be troublesome.

  The present invention has been made in view of the above problems, and an object of the present invention is to reduce the trouble of an operator who sets a system for generating a virtual viewpoint image.

  In order to solve the above problems, a control device according to the present invention has, for example, the following configuration. That is, the control device performs a setting process of a system that generates a virtual viewpoint image based on captured images obtained by capturing images from a plurality of directions, and holds a plurality of system setting information including a plurality of operation information. Holding control means for holding means, and setting means for performing system setting processing based on one system setting information of a plurality of system setting information held by the holding means, The operation information includes information on a specific area to which a plurality of imaging devices are directed, and the setting processing includes a setting processing of directing a plurality of cameras to the specific area.

  ADVANTAGE OF THE INVENTION According to this invention, the trouble of the operator which sets the system for producing | generating a virtual viewpoint image can be reduced.

FIG. 1 is a diagram for describing a configuration of an image processing system 100. FIG. 3 is a block diagram for describing a functional configuration of a camera adapter 120. It is a block diagram for explaining composition of image processing part 6130. FIG. 3 is a block diagram for describing a functional configuration of a front-end server 230. FIG. 14 is a block diagram for describing a configuration of a data input control unit 02120 of the front-end server 230. FIG. 3 is a block diagram for explaining a functional configuration of a database 250. FIG. 4 is a block diagram for explaining a functional configuration of a back-end server 270. FIG. 3 is a block diagram for describing a functional configuration of a virtual camera operation UI 330. FIG. 3 is a diagram for explaining a connection configuration of an end user terminal 190. FIG. 3 is a block diagram illustrating a functional configuration of an end user terminal 190. It is a flowchart for explaining the whole workflow. It is a flowchart for demonstrating the workflow before equipment installation. It is a flowchart for demonstrating the workflow at the time of equipment installation. It is a flowchart for demonstrating the workflow before imaging | photography. 6 is a flowchart for explaining a shooting confirmation workflow on the control station 310 side. 9 is a flowchart for explaining a user workflow at the time of shooting on the virtual camera operation UI 330 side. It is a sequence diagram for explaining the whole processing of at-installation calibration. 5 is a flowchart for explaining an operation of the front-end server 230 before shooting. 5 is a flowchart for explaining an operation of a database 250 before photographing. 5 is a flowchart for explaining the operation of the database 250 during shooting. It is a flowchart for demonstrating the process of the calibration at the time of installation. FIG. 9 is a sequence diagram for explaining a shooting start process. FIG. 9 is a sequence diagram for explaining a generation process of three-dimensional model information. It is a flow chart for explaining generation processing of three-dimensional model information. It is a flow chart for explaining generation processing of three-dimensional model information. It is a figure for explaining a fixation point group. FIG. 4 is a diagram for describing bypass transmission control. It is a figure for explaining bypass control. FIG. 3 is a diagram for explaining a data transmission flow. It is a flowchart for demonstrating the transmission data reduction process. It is a flowchart for explaining a file generation process. 13 is a flowchart for describing a process of writing a file to the database 250. 13 is a flowchart for describing a process of reading a file from a database 250. FIG. 4 is a diagram illustrating an example of a captured image. 9 is a flowchart for describing foreground / background separation. FIG. 7 is a sequence diagram for explaining a virtual camera image generation process. FIG. 3 is a diagram for describing a virtual camera. 9 is a flowchart for describing a live image generation process. It is a flow chart for explaining generation processing of a replay image. It is a flowchart for explaining selection of a virtual camera path. FIG. 9 is a diagram illustrating an example of a screen displayed by the end user terminal 190. 6 is a flowchart for explaining processing of an application management unit 10001 regarding manual operation. 6 is a flowchart for describing processing of an application management unit 10001 regarding automatic driving. It is a flowchart for explaining a rendering process. It is a flowchart for demonstrating the generation process of a foreground image. FIG. 9 is a diagram illustrating a setting list generated in a post-installation workflow. FIG. 7 is a sequence diagram for describing setting information change processing in a control station 310. 13 is a flowchart for describing a data reception process of the front-end server 230. FIG. 2 is a block diagram illustrating a hardware configuration of a camera adapter 120.

  A system in which a plurality of cameras and microphones are installed in a facility such as a stadium or a concert hall to capture and collect sound will be described with reference to the system configuration diagram in FIG. The image processing system 100 has a sensor system 110a-sensor system 110z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

  The controller 300 has a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management and parameter setting control for each block constituting the image processing system 100 through the networks 310a to 310c, 180a, 180b, and 170a to 170y. Here, the network may be GbE (Gigabit Ethernet) conforming to the IEEE standard, which is Ethernet (registered trademark, hereinafter abbreviated), or 10 GbE, or may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. The network is not limited to these, and may be another type of network.

  First, an operation of transmitting 26 sets of images and sounds of the sensor system 110a to the sensor system 110z from the sensor system 110z to the image computing server 200 will be described. In the image processing system 100 of the present embodiment, a sensor system 110a and a sensor system 110z are connected by a daisy chain.

  In the present embodiment, unless otherwise specified, the 26 sets of systems from the sensor system 110a to the sensor system 110z are described as the sensor system 110 without distinction. Similarly, the devices in each sensor system 110 will be described as a microphone 111, a camera 112, a camera platform 113, an external sensor 114, and a camera adapter 120 without distinction unless otherwise specified. Although the number of sensor systems is described as 26 sets, this is merely an example, and the number is not limited to this. Further, the plurality of sensor systems 110 do not have to have the same configuration, and for example, each may be configured by a device of a different model. Note that in the present embodiment, unless otherwise specified, the term image is described as including the concept of a moving image and a still image. That is, the image processing system 100 of the present embodiment can process both still images and moving images. Further, in the present embodiment, an example will be described in which the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual viewpoint sound, but the present invention is not limited to this. For example, the sound may not be included in the virtual viewpoint content. Further, for example, the sound included in the virtual viewpoint content may be sound collected by a microphone closest to the virtual viewpoint. In the present embodiment, for simplicity of description, description of audio is partially omitted, but it is assumed that both image and audio are basically processed.

  The sensor system 110a-sensor system 110z has one camera 112a-camera 112z, respectively. That is, the image processing system 100 includes a plurality of cameras 112 for photographing a subject from a plurality of directions. The plurality of cameras 112 will be described using the same reference numerals, but may differ in performance and model. The plurality of sensor systems 110 are connected by a daisy chain. With this connection form, it is possible to reduce the number of connection cables and to save labor for wiring work in increasing the resolution of a captured image to 4K or 8K and increasing the capacity of image data accompanying a higher frame rate. This is specified here.

  However, the present invention is not limited to this, and a star-type network configuration in which the sensor systems 110a to 110z are connected to the switching hub 180 and transmit and receive data between the sensor systems 110 via the switching hub 180 may be used as a connection form.

  FIG. 1 shows a configuration in which all of the sensor systems 110a to 110z are cascaded so as to form a daisy chain, but the present invention is not limited to this. For example, a plurality of sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy-chain connected in divided groups. Then, the camera adapter 120 which is the end of the division unit may be connected to the switching hub to input an image to the image computing server 200. Such a configuration is particularly effective in a stadium. For example, a case is considered in which a stadium is configured with a plurality of floors, and a sensor system 110 is provided for each floor. In this case, the input to the image computing server 200 can be performed for each floor or for every half of the stadium, so that the installation can be simplified even in a place where wiring for connecting all the sensor systems 110 by one daisy chain is difficult. The system can be made more flexible.

  Further, control of image processing in the image computing server 200 is switched according to whether there is one or more camera adapters 120 connected in a daisy chain and inputting an image to the image computing server 200. . That is, the control is switched according to whether the sensor system 110 is divided into a plurality of groups. When the number of camera adapters 120 for inputting an image is one, an image of the entire circumference of the stadium is generated while performing image transmission by daisy chain connection. Has been taken. That is, if the sensor system 110 is not divided into groups, synchronization can be achieved.

  However, when a plurality of camera adapters 120 perform image input, the delay from when an image is captured to when the image is input to the image computing server 200 may be different for each daisy chain lane (path). That is, when the sensor system 110 is divided into groups, the timing at which the image data of the entire circumference is input to the image computing server 200 may not be synchronized. Therefore, it is specified that the image processing server 200 needs to perform image processing in the subsequent stage while checking the aggregation of the image data by the synchronization control that waits until the image data on the entire circumference is prepared and synchronizes.

  In the present embodiment, the sensor system 110a has a microphone 111a, a camera 112a, a camera platform 113a, an external sensor 114a, and a camera adapter 120a. Note that the present invention is not limited to this configuration, and it is sufficient that at least one camera adapter 120a and one camera 112a or one microphone 111a are provided. Further, for example, the sensor system 110a may be configured with one camera adapter 120a and a plurality of cameras 112a, or may be configured with one camera 112a and a plurality of camera adapters 120a. That is, the plurality of cameras 112 and the plurality of camera adapters 120 in the image processing system 100 correspond to N to M (N and M are both integers of 1 or more). Further, the sensor system 110 may include devices other than the microphone 111a, the camera 112a, the camera platform 113a, and the camera adapter 120a. Further, the camera 112 and the camera adapter 120 may be integrally configured. Further, the front-end server 230 may have at least a part of the functions of the camera adapter 120. In the present embodiment, the sensor systems 110b to 110z will not be described because they have the same configuration as the sensor system 110a. The configuration is not limited to the same as the sensor system 110a, and each sensor system 110 may have a different configuration.

  The sound collected by the microphone 111a and the image captured by the camera 112a are transmitted to the camera adapter 120b of the sensor system 110b through the daisy chain 170a after being subjected to image processing described later in the camera adapter 120a. . Similarly, the sensor system 110b transmits the collected sound and the captured image to the sensor system 110c together with the image and the sound acquired from the sensor system 110a.

  By continuing the above-described operation, the image and the sound acquired by the sensor system 110a-sensor system 110z are transmitted from the sensor system 110z to the switching hub 180 by using 180b, and then transmitted to the image computing server 200.

  In this embodiment, the cameras 112a to 112z and the camera adapters 120a to 120z are separated from each other, but they may be integrated in the same housing. In that case, the microphones 111a to 111z may be built in the integrated camera 112 or may be connected to the outside of the camera 112.

  Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 according to the present embodiment processes data acquired from the sensor system 110z. The image computing server 200 includes a front-end server 230, a database 250 (hereinafter, also referred to as a DB), a back-end server 270, and a time server 290.

  The time server 290 has a function of distributing a time and synchronization signal, and distributes the time and synchronization signal to the sensor systems 110a to 110z via the switching hub 180. The camera adapters 120a to 120z that have received the time and the synchronization signal perform Genlock of the cameras 112a to 112z based on the time and the synchronization signal to perform image frame synchronization. That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112. Accordingly, since the image processing system 100 can generate the virtual viewpoint image based on the plurality of captured images captured at the same timing, it is possible to suppress a decrease in the quality of the virtual viewpoint image due to a shift in the capturing timing. In the present embodiment, the time server 290 manages the time synchronization of the plurality of cameras 112. However, the present invention is not limited to this. Each camera 112 or each camera adapter 120 performs the process for time synchronization independently. You may.

  The front-end server 230 reconstructs the segmented transmission packet from the image and sound obtained from the sensor system 110z and converts the data format, and then stores the data in the database 250 according to the camera identifier, data type, and frame number. Write.

  Next, the back-end server 270 receives the designation of the viewpoint from the virtual camera operation UI 330, reads the corresponding image and audio data from the database 250 based on the received viewpoint, and performs a rendering process to generate a virtual viewpoint image. I do.

  The configuration of the image computing server 200 is not limited to this. For example, at least two of the front-end server 230, the database 250, and the back-end server 270 may be integrally configured. Further, a plurality of at least one of the front-end server 230, the database 250, and the back-end server 270 may be included. Further, a device other than the above device may be included at an arbitrary position in the image computing server 200. Further, the end user terminal 190 and the virtual camera operation UI 330 may have at least a part of the functions of the image computing server 200.

  The rendered image is transmitted from the back-end server 270 to the end user terminal 190, and the user operating the end user terminal 190 can perform image browsing and audio viewing according to the designation of the viewpoint. That is, the back-end server 270 generates virtual viewpoint content based on the captured images (multiple viewpoint images) captured by the plurality of cameras 112 and the viewpoint information. More specifically, the back-end server 270, for example, based on the image data of a predetermined area extracted from the images captured by the plurality of cameras 112 by the plurality of camera adapters 120 and the viewpoint specified by the user operation, Generate content. Then, the back-end server 270 provides the generated virtual viewpoint content to the end user terminal 190. Details of the extraction of the predetermined area by the camera adapter 120 will be described later. Note that, in the present embodiment, the virtual viewpoint content is generated by the image computing server 200, and a description will be mainly given of a case where the virtual viewpoint content is generated by the back-end server 270. However, the present invention is not limited thereto, and the virtual viewpoint content may be generated by a device other than the back-end server 270 included in the image computing server 200, or may be generated by the controller 300 or the end-user terminal 190.

  The virtual viewpoint content in the present embodiment is a content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the designated viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on a result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily specified by the user. In addition, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to a viewpoint automatically designated by the device are also included in the virtual viewpoint image. In the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described, but audio data may not necessarily be included. Further, the back-end server 270 converts the virtual viewpoint image into, for example, H.264. After compression-encoding according to an encoding method such as H.264 or HEVC, the data may be transmitted to the end user terminal 190 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 190 without compression. In particular, the former for performing compression encoding assumes a smartphone or a tablet as the end user terminal 190, and the latter assumes a display capable of displaying an uncompressed image. That is, it is specified that the image format can be switched according to the type of the end user terminal 190. Further, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or another transmission method may be used.

  As described above, the image processing system 100 has three functional domains: the video collection domain, the data storage domain, and the video generation domain. The image collection domain includes the sensor systems 110-110z, the data storage domain includes the database 250, the front-end server 230 and the back-end server 270, and the image generation domain includes the virtual camera operation UI 330 and the end user terminal 190. The present invention is not limited to this configuration. For example, the virtual camera operation UI 330 can directly acquire an image from the sensor systems 110a to 110z. However, in the present embodiment, a method of arranging a data storage function in the middle rather than a method of directly acquiring an image from the sensor systems 110a to 110z is adopted. Specifically, the front-end server 230 converts the image data and audio data generated by the sensor systems 110a to 110z and the meta information of those data into a common schema and data type of the database 250. Thus, even if the camera 112 of the sensor system 110a-110z changes to a camera of another model, the changed difference can be absorbed by the front-end server 230 and registered in the database 250. This can reduce the risk that the virtual camera operation UI 330 will not operate properly when the camera 112 is changed to another model camera.

  Further, the virtual camera operation UI 330 is configured to access via the back-end server 270 without directly accessing the database 250. The common process related to the image generation process is performed by the back-end server 270, and the difference part of the application related to the operation UI is performed by the virtual camera operation UI 330. As a result, in the development of the virtual camera operation UI 330, it is possible to focus on the development for the UI operation device and the function request of the UI for operating the virtual viewpoint image to be generated. Further, the back-end server 270 can also add or delete common processing related to the image generation processing in response to a request from the virtual camera operation UI 330. This makes it possible to flexibly respond to the request of the virtual camera operation UI 330.

  As described above, in the image processing system 100, the virtual viewpoint image is generated by the back-end server 270 based on the image data based on the imaging by the plurality of cameras 112 for imaging the subject from a plurality of directions. Note that the image processing system 100 according to the present embodiment is not limited to the physical configuration described above, and may be configured logically. In the present embodiment, a technique for generating a virtual viewpoint image based on an image captured by the camera 112 will be described. For example, a virtual viewpoint image is generated based on an image generated by computer graphics or the like without using a captured image. In this case, the present embodiment can be applied.

  Next, a functional block diagram of each node (camera adapter 120, front-end server 230, database 250, back-end server 270, virtual camera operation UI 330, end-user terminal 190) in the system shown in FIG. 1 will be described.

  The functional blocks of the camera adapter 120 according to the present embodiment will be described with reference to FIG. The details of the flow of data between the functional blocks of the camera adapter 120 will be described later with reference to FIG.

  The camera adapter 120 includes a network adapter 06110, a transmission unit 06120, an image processing unit 06130, and an external device control unit 06140. The network adapter 06110 includes a data transmission / reception unit 06111 and a time control unit 06112.

  The data transmission / reception unit 06111 performs data communication with the other camera adapter 120, the front end server 230, the time server 290, and the control station 310 via the daisy chain 170, the network 291, and the network 310a. For example, the data transmitting / receiving unit 06111 outputs the foreground image and the background image separated by the foreground / background separation unit 06131 from the image captured by the camera 112 to another camera adapter 120. The output destination camera adapter 120 is the next camera adapter 120 in a predetermined order according to the processing of the data routing processing unit 06122 among the camera adapters 120 in the image processing system 100. When each camera adapter 120 outputs the foreground image and the background image, a virtual viewpoint image is generated based on the foreground image and the background image captured from a plurality of viewpoints. Note that there may be a camera adapter 120 that outputs a foreground image separated from a captured image and does not output a background image.

  The time control unit 06112 conforms to, for example, the Ordinary Clock of the IEEE 1588 standard, and performs a function of storing a time stamp of data transmitted and received between the time server 290 and time synchronization with the time server 290. Note that the present invention is not limited to the IEEE 1588, and the time synchronization with the time server may be realized by another EtherAVB standard or a unique protocol. In the present embodiment, a NIC (Network Interface Card) is used as the network adapter 06110. However, the present invention is not limited to the NIC, and other similar Interfaces may be used. IEEE 1588 has been updated as a standard such as IEEE 1588-2002 and IEEE 1588-2008, and the latter is also called PTPv2 (Precision Time Protocol Version 2).

  The transmission unit 06120 has a function of controlling data transmission to the switching hub 180 and the like via the network adapter 06110, and includes the following functional units.

  The data compression / expansion unit 06121 performs a function of applying a predetermined compression method, a compression rate, and a frame rate to data transmitted / received via the data transmission / reception unit 06111, and a function of expanding the compressed data. have.

  The data routing processing unit 06122 determines the routing destination of the data received by the data transmission / reception unit 06111 and the data processed by the image processing unit 06130 by using data stored in a data routing information storage unit 06125 described later. Further, it has a function of transmitting data to the determined routing destination. As a routing destination, the camera adapter 120 corresponding to the camera 112 focused on the same gazing point is preferable in performing image processing because the image frame correlation between the cameras 112 is high. The order of the camera adapters 120 that output the foreground image and the background image in the relay format in the image processing system 100 is determined according to the determination by the data routing processing unit 06122 of each of the plurality of camera adapters 120.

  The time synchronization control unit 06123 conforms to the PTP (Precision Time Protocol) of the IEEE1588 standard, and has a function of performing processing related to time synchronization with the time server 290. It should be noted that the time synchronization is not limited to the PTP but may be performed using another similar protocol.

  The image / audio transmission processing unit 06124 has a function of creating a message for transferring image data or audio data to another camera adapter 120 or the front-end server 230 via the data transmission / reception unit 06111. The message includes image data or audio data, and meta information of each data. The meta information according to the present embodiment includes a time code or a sequence number at the time of capturing an image or sampling a sound, a data type, an identifier indicating an individual camera 112 or a microphone 111, and the like. The image data or audio data to be transmitted may be data compressed by the data compression / expansion unit 06121. In addition, the image / audio transmission processing unit 06124 receives a message from another camera adapter 120 via the data transmission / reception unit 06111. Then, according to the data type included in the message, the data information fragmented to the packet size specified by the transmission protocol is restored to image data or audio data. If the data is compressed when the data is restored, the data compression / decompression unit 06121 performs a decompression process.

  The data routing information holding unit 06125 has a function of holding address information for determining a destination of data transmitted / received by the data transmitting / receiving unit 06111. The routing method will be described later.

  The image processing unit 06130 has a function of processing image data captured by the camera 112 and image data received from another camera adapter 120 under the control of the camera control unit 06141, and includes the following functional units. I have.

  The foreground / background separation unit 06131 has a function of separating image data captured by the camera 112 into a foreground image and a background image. That is, each of the plurality of camera adapters 120 operates as an image processing device that extracts a predetermined area from an image captured by the corresponding camera 112 among the plurality of cameras 112. The predetermined area is, for example, a foreground image obtained as a result of object detection with respect to the captured image. With this extraction, the foreground / background separation unit 06131 separates the captured image into a foreground image and a background image. The object is, for example, a person. However, the object may be a specific person (such as a player, a manager, and / or a referee), or may be an object with a predetermined image pattern, such as a ball or a goal. Further, a moving object may be detected as an object. By separating and processing a foreground image including an important object such as a person and a background region not including such an object, an image of a portion corresponding to the above object of the virtual viewpoint image generated in the image processing system 100 Quality can be improved. Further, since the foreground and the background are separated by each of the plurality of camera adapters 120, the load on the image processing system 100 including the plurality of cameras 112 can be distributed. The predetermined area is not limited to the foreground image, and may be, for example, a background image.

  The three-dimensional model information generation unit 06132 uses the foreground image separated by the foreground / background separation unit 06131 and the foreground image received from the other camera adapter 120, and uses, for example, the principle of a stereo camera to generate image information related to the three-dimensional model. Is generated.

  The calibration control unit 06133 has a function of acquiring image data necessary for calibration from the camera 112 via the camera control unit 06141 and transmitting the acquired image data to the front-end server 230 that performs arithmetic processing related to calibration. . The calibration according to the present embodiment is a process of associating parameters relating to the respective cameras 112 with each other. As the calibration, for example, a process of adjusting the world coordinate system held by each installed camera 112 so as to match, a color correction process for suppressing a variation in color of each camera 112, and the like are performed. The specific processing content of the calibration is not limited to this. Further, in the present embodiment, the arithmetic processing related to the calibration is performed by the front-end server 230, but the node that performs the arithmetic processing is not limited to the front-end server 230. For example, the arithmetic processing may be performed by another node such as the control station 310 or the camera adapter 120 (including another camera adapter 120). Further, the calibration control unit 06133 has a function of performing calibration (dynamic calibration) during photographing on image data acquired from the camera 112 via the camera control unit 06141 according to preset parameters. are doing.

  The external device control unit 06140 has a function of controlling devices connected to the camera adapter 120, and includes the following functional blocks.

  The camera control unit 06141 is connected to the camera 112 and has a function of controlling the camera 112, obtaining a captured image, providing a synchronization signal, setting a time, and the like. The control of the camera 112 includes, for example, setting and reference of shooting parameters (eg, setting of the number of pixels, color depth, frame rate, and white balance), and the state of the camera 112 (while shooting, stopping, synchronizing, and errors). Acquisition, start and stop of shooting, focus adjustment, and the like. In this embodiment, focus adjustment is performed via the camera 112. However, when a detachable lens is attached to the camera 112, the camera adapter 120 is connected to the lens and directly adjusts the lens. Is also good. Further, the camera adapter 120 may perform lens adjustment such as zooming via the camera 112. The synchronization signal is provided by providing the camera 112 with the shooting timing (control clock) using the time synchronized by the time synchronization control unit 06123 with the time server 290. The time setting is performed by providing the time synchronized by the time synchronization control unit 06123 with the time server 290 with a time code conforming to, for example, the SMPTE12M format. Thus, the provided time code is added to the image data received from the camera 112. The format of the time code is not limited to SMPTE12M, but may be another format. Further, the camera control unit 06141 may not provide the time code to the camera 112, and may add the time code to the image data received from the camera 112.

  The microphone control unit 06142 is connected to the microphone 111 and has a function of controlling the microphone 111, starting and stopping sound collection, acquiring collected sound data, and the like. The control of the microphone 111 is, for example, gain adjustment and state acquisition. Further, similarly to the camera control unit 06141, it provides the microphone 111 with the timing and time code for audio sampling. As clock information serving as a timing of audio sampling, time information from the time server 290 is converted into, for example, a 48 KHz word clock and supplied to the microphone 111.

  The camera platform controller 06143 is connected to the camera platform 113 and has a function of controlling the camera platform 113. The control of the camera platform 113 includes, for example, pan / tilt control and status acquisition.

  The sensor control unit 06144 has a function of connecting to the external sensor 114 and acquiring sensor information sensed by the external sensor 114. For example, when a gyro sensor is used as the external sensor 114, information representing vibration can be obtained. Then, using the vibration information acquired by the sensor control unit 06144, the image processing unit 06130 can generate an image in which the influence of the vibration of the camera 112 is reduced prior to the processing in the foreground / background separation unit 06131. . The vibration information is used, for example, when the image data of the 8K camera is cut out in a size smaller than the original 8K size in consideration of the vibration information, and alignment with the image of the camera 112 installed adjacently is performed. . Thereby, even if the frame vibration of the building propagates to each camera at a different frequency, the positioning is performed by the function provided in the camera adapter 120. As a result, it is possible to generate image data in which the influence of vibration is reduced (electronically damped) by the image processing, and the effect of reducing the processing load of the image computing server 200 for the number of cameras 112 is reduced. can get. Note that the sensor of the sensor system 110 is not limited to the external sensor 114, and the same effect can be obtained even if the sensor is built in the camera adapter 120.

  FIG. 3 is a functional block diagram of the image processing unit 06130 inside the camera adapter 120. The calibration control unit 06133 performs a color correction process on the input image to suppress a variation in color among the cameras, and a shake for stabilizing the image by reducing the image blur caused by the camera vibration. Correction processing (electronic anti-vibration processing) and the like are performed.

  The functional blocks of the foreground / background separation unit 06131 will be described. The foreground separation unit 05001 performs a foreground image separation process on the image data that has been aligned with respect to the image of the camera 112 by comparing the image data with the background image 05002.

  The background update unit 05003 generates a new background image using the image in which the position of the background image 05002 and the camera 112 have been aligned, and updates the background image 05002 with the new background image.

  The background extraction unit 05004 performs control to extract a part of the background image 05002.

Here, the function of the three-dimensional model information generation unit 06132 will be described.
The three-dimensional model processing unit 05005 uses the foreground image separated by the foreground separation unit 05001 and the foreground image of another camera 112 received via the transmission unit 06120 to form a three-dimensional model based on, for example, the principle of a stereo camera. Related image information is sequentially generated.
The other camera foreground receiving unit 05006 receives the foreground image whose foreground and background have been separated by the other camera adapter 120.

  The camera parameter receiving unit 05007 receives camera-specific internal parameters (focal length, image center, lens distortion parameters, and the like) and external parameters (rotation matrix, position vector, and the like) representing the position and orientation of the camera. These parameters are information obtained by a calibration process described later, and are transmitted and set from the control station 310 to the target camera adapter 120. Next, the three-dimensional model processing unit 05005 generates three-dimensional model information using the camera parameter receiving unit 05007 and the other camera foreground receiving unit 05006.

  FIG. 4 is a diagram showing functional blocks of the front-end server 230. The control unit 02110 is configured by a CPU, a DRAM, a storage medium such as an HDD or a NAND memory storing program data and various data, and hardware such as Ethernet. Then, it controls each functional block of the front-end server 230 and the entire system of the front-end server 230. In addition, mode control is performed to switch operation modes such as a calibration operation, a preparation operation before shooting, and an operation during shooting. Further, the control unit 310 receives a control instruction from the control station 310 through the Ethernet, switches between modes, inputs and outputs data, and the like. Also, stadium CAD data (stadium shape data) is obtained from the control station 310 via the network, and the stadium CAD data is transmitted to the CAD data storage unit 02135 and the photographing data file generation unit 02180. The stadium CAD data (stadium shape data) in the present embodiment is three-dimensional data indicating the shape of the stadium, and may be any data representing a mesh model or other three-dimensional shapes, and is not limited to the CAD format.

  The data input control unit 02120 is network-connected to the camera adapter 120 via a communication path such as Ethernet and the switching hub 180. Then, the data input control unit 02120 acquires the foreground image, the background image, the three-dimensional model of the subject, the audio data, and the camera calibration image data from the camera adapter 120 through the network. Here, the foreground image is image data based on the foreground area of the captured image for generating the virtual viewpoint image, and the background image is image data based on the background area of the captured image. The camera adapter 120 specifies a foreground area and a background area in accordance with a result of a predetermined object detection process performed on an image captured by the camera 112, and generates a foreground image and a background image. The predetermined object is, for example, a person. Note that the predetermined object may be a specific person (a player, a manager, and / or a referee, for example). In addition, the predetermined object may include an object such as a ball or a goal in which an image pattern is predetermined. Further, a moving object may be detected as a predetermined object.

  In addition, the data input control unit 02120 transmits the acquired foreground image and background image to the data synchronization unit 02130, and transmits camera calibration image data to the calibration unit 02140. Further, the data input control unit 02120 has a function of performing compression / decompression of received data, data routing processing, and the like. Further, both the control unit 02110 and the data input control unit 02120 have a communication function using a network such as Ethernet, but the communication function may be shared by these. In that case, a method may be used in which the data input control unit 02120 receives an instruction by a control command from the control station 310 or stadium CAD data and sends the data to the control unit 02110.

  The data synchronization unit 02130 temporarily stores the data acquired from the camera adapter 120 on the DRAM, and buffers the data until all foreground images, background images, audio data, and three-dimensional model data are obtained. Note that the foreground image, the background image, the audio data, and the three-dimensional model data are collectively referred to as photographing data hereinafter. The shooting data is provided with meta information such as routing information, time code information (time information), and a camera identifier, and the data synchronization unit 02130 confirms the attribute of the data based on the meta information. Accordingly, the data synchronizer 02130 determines that the data is at the same time or the like, and confirms that the data is complete. This is because the order in which network packets are received for data transferred from each camera adapter 120 via the network is not guaranteed, and data must be buffered until data necessary for file generation is completed. When the data is completed, the data synchronization unit 02130 transmits the foreground image and the background image to the image processing unit 02150, the three-dimensional model data to the three-dimensional model combining unit 02160, and the audio data to the photographing data file generation unit 02180. Note that the data to be aligned here is data necessary for generating a file in a shooting data file generation unit 02180 described later. Further, the background image may be shot at a frame rate different from that of the foreground image. For example, when the frame rate of the background image is 1 fps, one background image is acquired every second. Therefore, as for the time during which the background image is not acquired, all data may be collected without the background image. . In addition, when a predetermined time has elapsed and the data is not complete, the data synchronization unit 02130 notifies the database 250 of information indicating that the data is not available. Then, when storing the data, the database 250 at the subsequent stage stores information indicating the lack of the data together with the camera number and the frame number. Accordingly, whether or not a desired image can be formed from the captured images of the cameras 112 collected in the database 250 can be automatically notified before rendering according to a viewpoint instruction from the virtual camera operation UI 330 to the back-end server 270. It becomes possible. As a result, the visual load on the operator of the virtual camera operation UI 330 can be reduced.

  The CAD data storage unit 02135 stores the three-dimensional data indicating the stadium shape received from the control unit 02110 in a storage medium such as a DRAM, an HDD, or a NAND memory. Then, the stadium shape data stored when the request for the stadium shape data is received is transmitted to the image combining unit 02170.

  The calibration unit 02140 performs a camera calibration operation, and sends camera parameters obtained by the calibration to a non-shooting data file generation unit 02185 described later. At the same time, the camera parameters are held in its own storage area, and the camera parameters information is provided to a three-dimensional model combining unit 02160 described later.

  The image processing unit 02150 performs processing such as matching of colors and luminance values between cameras with respect to a foreground image and a background image, development processing when RAW image data is input, and correction of camera lens distortion. . Then, the foreground image subjected to the image processing is transmitted to the photographing data file generation unit 02180, and the background image is transmitted to 02170.

  The three-dimensional model combining unit 02160 combines the three-dimensional model data at the same time acquired from the camera adapter 120 using the camera parameters generated by the calibration unit 02140. Then, using a method called VisualHull, three-dimensional model data of the foreground image in the entire stadium is generated. The generated three-dimensional model is transmitted to the imaging data file generation unit 02180.

  The image combining unit 02170 acquires a background image from the image processing unit 02150, acquires three-dimensional shape data (stadium shape data) of the stadium from the CAD data storage unit 02135, and obtains a background image corresponding to the coordinates of the acquired three-dimensional shape data of the stadium. Identify the location of When the position of each of the background images with respect to the coordinates of the three-dimensional shape data of the stadium can be specified, the background images are combined into one background image. The back-end server 270 may create the three-dimensional shape data of the background image.

  The photographing data file generation unit 02180 has combined the audio data from the data synchronization unit 02130, the foreground image from the image processing unit 02150, the three-dimensional model data from the three-dimensional model combining unit 02160, and the three-dimensional shape from the image combining unit 02170. Get the background image. The obtained data is output to the DB access control unit 02190. Here, the photographing data file generation unit 02180 outputs these data in association with each other based on the time information. However, some of these data may be output in association with each other. For example, the photographing data file generation unit 02180 outputs the foreground image and the background image in association with each other based on the time information of the foreground image and the time information of the background image. Further, for example, the photographing data file generation unit 02180 associates the foreground image, the background image, and the three-dimensional model data based on the time information of the foreground image, the time information of the background image, and the time information of the three-dimensional model data. Output. Note that the photographing data file generation unit 02180 may output the associated data as a file for each type of data, or may collectively output a plurality of types of data for each time indicated by the time information. Is also good. The image data thus associated is output to the database 250 from the front-end server 230 as an information processing device that performs the association, so that the back-end server 270 generates a foreground image and a background image corresponding to time information. Can generate a virtual viewpoint image.

  When the frame rates of the foreground image and the background image acquired by the data input control unit 02120 are different, it is difficult for the photographing data file generation unit 02180 to always output the foreground image and the background image at the same time in association with each other. Therefore, the photographing data file generation unit 02180 associates the time information of the foreground image with the background image having the time information having a relationship based on a predetermined rule, and outputs the image. Here, the background image having the time information that is related to the time information of the foreground image based on a predetermined rule is, for example, the time information closest to the time information of the foreground image among the background images acquired by the photographing data file generation unit 02180. FIG. In this way, by associating the foreground image and the background image based on the predetermined rule, even when the frame rates of the foreground image and the background image are different, the virtual viewpoint image is obtained from the foreground image and the background image taken at close times. Can be generated. The method of associating the foreground image with the background image is not limited to the above. For example, a background image having time information that is in a relationship based on a predetermined rule with the time information of the foreground image is a background image that is an acquired background image and has time information corresponding to a time earlier than the foreground image. The background image having the time information closest to the time information of the foreground image may be used. According to this method, the associated foreground image and background image can be output with low delay without waiting for the acquisition of the background image having a lower frame rate than the foreground image. Further, the background image having the time information in a relationship based on the predetermined rule with the time information of the foreground image is an acquired background image, among the background images having time information corresponding to a time later than the foreground image, A background image having time information closest to the time information of the foreground image may be used.

  The non-shooting data file generation unit 02185 acquires the camera parameters from the calibration unit 02140, and the three-dimensional shape data of the stadium from the control unit 02110, shapes the data in accordance with the file format, and transmits the data to the DB access control unit 02190. Note that the camera parameters or stadium shape data, which are data input to the non-shooting data file generation unit 02185, are individually formed according to the file format. That is, when receiving either one of the data, the non-shooting data file generation unit 02185 individually transmits the data to the DB access control unit 02190.

  The DB access control unit 02190 is connected to the database 250 so that high-speed communication can be performed by InfiniBand or the like. Then, the files received from the imaging data file generation unit 02180 and the non-imaging data file generation unit 02185 are transmitted to the database 250. In the present embodiment, the photographing data associated with the photographing data file generation unit 02180 based on the time information is transmitted via the DB access control unit 02190 to the database 250 which is a storage device connected to the front-end server 230 via a network. Output. However, the output destination of the associated shooting data is not limited to this. For example, the front-end server 230 outputs photographing data associated with the time information to the back-end server 270 that is an image generation device that is connected to the front-end server 230 via a network and generates a virtual viewpoint image. You may. Further, the data may be output to both the database 250 and the back-end server 270.

  In this embodiment, the front-end server 230 associates the foreground image with the background image. However, the present invention is not limited to this, and the database 250 may associate the foreground image with the background image. For example, the database 250 acquires a foreground image and a background image having time information from the front-end server 230. Then, the database 250 may associate the foreground image with the background image based on the time information of the foreground image and the time information of the background image, and output the image to a storage unit included in the database 250.

  A functional block diagram of the data input control unit 02120 of the front-end server 230 will be described with reference to FIG.

  The data input control unit 02120 includes a server network adapter 06210, a server transmission unit 06220, and a server image processing unit 06230.

  The server network adapter 06210 has a server data reception unit 062211, and has a function of receiving data transmitted from the camera adapter 120.

  The server transmission unit 06220 has a function of performing processing on the data received from the server data reception unit 06221, and includes the following functional units.

  The server data decompression unit 06221 has a function of decompressing compressed data.

  The server data routing processing unit 06222 determines a data transfer destination based on routing information such as an address held by a server data routing information holding unit 06224 described later, and transfers the data received from the server data receiving unit 06221.

  The server image transmission processing unit 06223 receives the message from the camera adapter 120 via the server data reception unit 06221, and restores the fragmented data into image data or audio data according to the data type included in the message. If the restored image data and audio data are compressed, the server data decompression unit 06221 performs decompression processing.

  The server data routing information holding unit 06224 has a function of holding address information for determining the destination of the data received by the server data receiving unit 062211. The routing method will be described later.

  The server image processing unit 06230 has a function of performing processing relating to image data or audio data received from the camera adapter 120. The processing content includes, for example, camera number, shooting time of image frame, image size, image format, and attribute information of image coordinates according to the data entity of the image data (foreground image, background image, and three-dimensional model information). For example, a shaping process into a format to which a format is added.

FIG. 6 is a diagram showing functional blocks of the database 250. The control unit 02410 is configured by a CPU, a DRAM, a storage medium such as an HDD or a NAND memory that stores program data and various data, and hardware such as Ethernet. Then, it controls each functional block of the database 250 and the entire system of the database 250.
The data input unit 02420 receives a file of image data or non-image data from the front-end server 230 by high-speed communication such as InfiniBand. The received file is sent to the cache 02440. It also reads the meta information of the received image data and creates a database table based on the time code information, routing information, camera identifier, and other information recorded in the meta information so that the acquired data can be accessed. I do.

  The data output unit 02430 determines whether the data requested from the back-end server 270 is stored in a cache 02440, a primary storage 02450, or a secondary storage 02460 described later. Then, by high-speed communication such as InfiniBand, the data is read from the saved destination and transmitted to the back-end server 270.

  The cache 02440 has a storage device such as a DRAM capable of realizing high-speed input / output throughput, and stores photographed data and non-photographed data acquired from the data input unit 02420 in the storage device. The stored data is held in a fixed amount, and when data exceeding the fixed amount is input, old data is written out to the primary storage 02450 as needed, and the written data is overwritten by new data. Here, the data stored in the cache 02440 by a fixed amount is photographing data for at least one frame. Thereby, when performing the image rendering processing in the back-end server 270, it is possible to minimize the throughput in the database 250 and to render the latest image frame continuously with low delay. Here, in order to achieve the above-mentioned object, the cached data needs to include a background image. Therefore, when photographing data of a frame having no background image is cached, the background image in the cache is not updated and is held in the cache as it is. The capacity of the DRAM that can be cached is determined by a cache frame size preset in the system or an instruction from the control station. The non-photographed data is copied to the primary storage immediately because the input / output frequency is low and high throughput is not required before the game. The cached data is read by the data output unit 02430.

  The primary storage 02450 is configured by connecting storage media such as SSDs in parallel, etc., and has a high speed such that a large amount of data can be written from the data input unit 02420 and data can be read from the data output unit 02430 at the same time. Then, the data stored in the cache 02440 is written out to the primary storage 02450 in order from the oldest one.

  The secondary storage 02460 is composed of an HDD, a tape medium, or the like, and a large capacity is more important than high-speed performance, and is required to be a medium that is inexpensive and suitable for long-term storage as compared with the primary storage. After the photographing is completed, the data stored in the primary storage 02450 is written out to the secondary storage 02460 as a backup of the data.

  FIG. 7 shows a configuration of the back-end server 270 according to the present embodiment. The back-end server 270 includes a data receiving unit 03001, a background texture pasting unit 03002, a foreground texture determination unit 03003, a texture boundary color matching unit 03004, a virtual viewpoint foreground image generation unit 03005, and a rendering unit 03006. Furthermore, it has a virtual viewpoint voice generation unit 03007, a synthesis unit 03008, an image output unit 03909, a foreground object determination unit 03010, a request list generation unit 03011, a request data output unit 03012, and a rendering mode management unit 03014.

  The data receiving unit 03001 receives data transmitted from the database 250 and the controller 300. The database 250 also receives three-dimensional data indicating the shape of the stadium (stadium shape data), a foreground image, a background image, a three-dimensional model of the foreground image (hereinafter referred to as a foreground three-dimensional model), and audio.

  Further, the data receiving unit 03001 receives virtual camera parameters output from the controller 300 as a specifying device that specifies a viewpoint related to generation of a virtual viewpoint image. The virtual camera parameters are data representing the position and orientation of the virtual viewpoint, and for example, a matrix of external parameters and a matrix of internal parameters are used.

  Note that data acquired by the data receiving unit 03001 from the controller 300 is not limited to virtual camera parameters. For example, the information output from the controller 300 includes at least one of a method of designating a viewpoint, information specifying an application operated by the controller, identification information of the controller 300, and identification information of a user who uses the controller 300. May be. Further, the data receiving unit 03001 may acquire, from the end user terminal 190, information similar to the above information output from the controller 300. Further, the data receiving unit 03001 may acquire information on the plurality of cameras 112 from an external device such as the database 250 or the controller 300. The information on the plurality of cameras 112 is, for example, information on the number of the plurality of cameras 112, information on the operation state of the plurality of cameras 112, and the like. The operation state of the camera 112 includes, for example, at least one of a normal state, a failure state, a standby state, a start state, and a restart state of the camera 112.

The background texture pasting unit 03002 pastes a background image as a texture to the three-dimensional space shape indicated by the background mesh model (stadium shape data) acquired from the background mesh model management unit 03013. Thus, the background texture pasting unit 03002 generates a textured background mesh model. The mesh model is data expressing a three-dimensional space shape such as CAD data by a set of surfaces. The texture is an image to be pasted to express the texture of the surface of the object.
The foreground texture determination unit 03003 determines the texture information of the foreground three-dimensional model from the foreground image and the foreground three-dimensional model group.

  The foreground texture boundary color matching unit 03004 performs color matching of the texture boundary from the texture information of each foreground three-dimensional model and each three-dimensional model group, and generates a colored foreground three-dimensional model group for each foreground object.

  The virtual viewpoint foreground image generation unit 03005 performs perspective transformation of the foreground image group based on the virtual camera parameters so that the foreground image group looks like the virtual viewpoint. The rendering unit 03006 renders the background image and the foreground image based on the generation method used for generating the virtual viewpoint image determined by the rendering mode management unit 03014, and generates a full-view virtual viewpoint image.

  In the present embodiment, two rendering modes of model-based rendering (Model-Based Rendering: MBR) and image-based (Image-Based Rendering: IBR) are used as a method of generating a virtual viewpoint image.

  MBR is a method of generating a virtual viewpoint image using a three-dimensional model generated based on a plurality of captured images of a subject captured in a plurality of directions. Specifically, using the three-dimensional shape (model) of the target scene obtained by a three-dimensional shape restoration method such as a volume intersection method or Multi-View-Stereo (MVS), the scene appearance from a virtual viewpoint is used. This is a technology for generating images.

  IBR is a technique for generating a virtual viewpoint image that reproduces the appearance from a virtual viewpoint by deforming and combining an input image group obtained by photographing a target scene from a plurality of viewpoints. In the present embodiment, when the IBR is used, the virtual viewpoint image is generated based on one or a plurality of captured images smaller than the plurality of captured images for generating the three-dimensional model using the MBR.

  When the rendering mode is MBR, a full-view model is generated by combining the background mesh model and the foreground three-dimensional model group generated by the foreground texture boundary color matching unit 03004, and a virtual viewpoint image is generated from the full-view model.

  When the rendering mode is IBR, a background image viewed from a virtual viewpoint is generated based on the background texture model, and a foreground image generated by the virtual viewpoint foreground image generation unit 03005 is synthesized therewith to generate a virtual viewpoint image. You.

  Note that the rendering unit 03006 may use a rendering method other than MBR and IBR. Further, the generation method of the virtual viewpoint image determined by the rendering mode management unit 03014 is not limited to the rendering method, and the rendering mode management unit 03014 may determine a processing method other than the rendering for generating the virtual viewpoint image. . The rendering mode management unit 03014 determines a rendering mode as a generation method used for generating a virtual viewpoint image, and holds the determination result.

  In the present embodiment, the rendering mode management unit 03014 determines a rendering mode to be used from a plurality of rendering modes. This determination is made based on the information acquired by the data receiving unit 03001. For example, when the number of cameras specified from the acquired information is equal to or less than the threshold, the rendering mode management unit 03014 determines the generation method used for generating the virtual viewpoint image to be IBR. On the other hand, if the number of cameras is larger than the threshold, the generation method is determined to be MBR. Accordingly, when the number of cameras is large, a virtual viewpoint image is generated using MBR, so that the specifiable range of the viewpoint is widened. Further, when the number of cameras is small, by using IBR, it is possible to avoid a decrease in image quality of the virtual viewpoint image due to a decrease in accuracy of the three-dimensional model when MBR is used. Further, for example, the generation method may be determined based on the allowable processing delay time from photographing to image output. If priority is given to the degree of freedom of the viewpoint even if the delay time is long, MBR is used, and if short delay time is required, IBR is used. For example, when the data receiving unit 03001 acquires information indicating that the controller 300 or the end user terminal 190 can specify the height of the viewpoint, the generation method used for generating the virtual viewpoint image is determined to be MBR. I do. Thus, it is possible to prevent the user's request for changing the height of the viewpoint from being unacceptable due to the generation method being IBR. As described above, by determining the generation method of the virtual viewpoint image according to the situation, it is possible to generate the virtual viewpoint image by the appropriately determined generation method. In addition, it is specified that a system can be flexibly configured by making a configuration in which a plurality of rendering modes can be switched according to a request, and the present embodiment can be applied to subjects other than a stadium. Keep it.

  Note that the rendering mode held by the rendering mode management unit 03014 may be a method preset in the system. Further, the user operating the virtual camera operation UI 330 or the end user terminal 190 may be able to arbitrarily set the rendering mode.

  The virtual viewpoint sound generation unit 03007 generates sounds (sound group) that can be heard at the virtual viewpoint based on the virtual camera parameters. The synthesizing unit 03008 synthesizes the image group generated by the rendering unit 03006 and the audio generated by the virtual viewpoint audio generation unit 03007 to generate virtual viewpoint content.

  The image output unit 03909 outputs virtual viewpoint content to the controller 300 and the end user terminal 190 using Ethernet. However, transmission means to the outside is not limited to Ethernet, and signal transmission means such as SDI, DisplayPort, and HDMI (registered trademark) may be used. Note that the back-end server 270 may output a virtual viewpoint image that does not include audio and is generated by the rendering unit 03006.

  A foreground object determination unit 03010 determines a foreground object group to be displayed from the virtual camera parameters and position information of the foreground object indicating the position of the foreground object included in the foreground three-dimensional model, and outputs a foreground object list. I do. That is, the foreground object determination unit 03010 performs a process of mapping the image information of the virtual viewpoint to the physical camera 112. This virtual viewpoint has a different mapping result depending on the rendering mode determined by the rendering mode management unit 03014. Therefore, it is specified that a control unit that determines a plurality of foreground objects is provided in the foreground object determination unit 03010 and performs control in conjunction with the rendering mode.

  The request list generation unit 03011 generates a request list for requesting the database 250 for the foreground image group and the foreground three-dimensional model group corresponding to the foreground object list at the designated time, and the background image and audio data. For the foreground object, data selected in consideration of the virtual viewpoint is required from the database 250, but for the background image and audio data, all data related to the frame is required. After the start of the back-end server 270, a background mesh model request list is generated until the background mesh model is acquired.

  The request data output unit 03012 outputs a data request command to the database 250 based on the input request list. The background mesh model management unit 03013 stores the background mesh model received from the database 250.

  In the present embodiment, a case will be mainly described in which the back-end server 270 determines both the generation method of the virtual viewpoint image and the generation of the virtual viewpoint image, but is not limited thereto. That is, the information processing apparatus that has determined the generation method may output data according to the determination result. For example, the front-end server 230 determines a generation method used for generating a virtual viewpoint image based on information regarding the plurality of cameras 112 and information output from a device that specifies a viewpoint related to generation of the virtual viewpoint image. You may. Then, the front-end server 230 outputs the image data based on the image captured by the camera 112 and the information indicating the determined generation method to at least one of a storage device such as the database 250 and an image generation device such as the back-end server 270. You may. In this case, for example, the back-end server 270 generates a virtual viewpoint image based on the information indicating the generation method output by the front-end server 230. When the front-end server 230 determines the generation method, it is possible to reduce the processing load caused by the database 250 and the back-end server 270 processing data for image generation in a method different from the determined method. On the other hand, when the back-end server 270 determines the generation method as in the present embodiment, the database 250 holds data that can correspond to the plurality of generation methods, so that the plurality of virtual viewpoint images corresponding to the plurality of generation methods respectively. Can be generated.

  FIG. 8 is a block diagram illustrating a functional configuration of the virtual camera operation UI 330. The virtual camera 08001 will be described with reference to FIG. The virtual camera 08001 is a virtual camera that can shoot at a different viewpoint from any of the cameras 112 installed. That is, the virtual viewpoint image generated in the image processing system 100 is a captured image of the virtual camera 08001. In FIG. 37A, each of a plurality of sensor systems 110 arranged on the circumference has a camera 112. For example, by generating a virtual viewpoint image, it is possible to generate an image as if it were taken by a virtual camera 08001 near a soccer goal. A virtual viewpoint image which is a captured image of the virtual camera 08001 is generated by performing image processing on images of the plurality of cameras 112 installed. An operator (user) can obtain a captured image from a free viewpoint by operating the position of the virtual camera 08001 and the like.

  The virtual camera operation UI 330 has a virtual camera management unit 08130 and an operation UI unit 08120. These may be mounted on the same device, or may be separately mounted on a device serving as a server and a device serving as a client. For example, in the virtual camera operation UI 330 used by the broadcasting station, the virtual camera management unit 08130 and the operation UI unit 08120 may be mounted on a workstation in the relay van. Further, for example, the same function may be realized by mounting the virtual camera management unit 08130 on a web server and mounting the operation UI unit 08120 on the end user terminal 190.

  The virtual camera operation unit 08101 receives and processes an operation performed by the operator on the virtual camera 08001, that is, an instruction from a user to specify a viewpoint related to generation of a virtual viewpoint image. The operation contents of the operator include, for example, change of position (movement), change of posture (rotation), change of zoom magnification, and the like. The operator uses an input device such as a joystick, a jog dial, a touch panel, a keyboard, and a mouse to operate the virtual camera 08001. The correspondence between the input by each input device and the operation of the virtual camera 08001 is determined in advance. For example, the “W” key on the keyboard is associated with an operation of moving the virtual camera 08001 forward by one meter. Further, the operator can operate the virtual camera 08001 by designating the trajectory. For example, the operator specifies a trajectory of the virtual camera 08001 rotating on a circle around the goal post by touching the touchpad in a circular manner. The virtual camera 08001 moves around the goal post along the designated trajectory. At this time, the posture may be automatically changed so that the virtual camera 08001 always faces the goal post. The virtual camera operation unit 08101 can be used for generating a live image and a replay image. When generating a replay image, an operation of designating a time in addition to the position and orientation of the camera is performed. In the replay image, for example, it is also possible to stop the time and move the virtual camera 08001.

  The virtual camera parameter deriving unit 08102 derives virtual camera parameters indicating the position, posture, and the like of the virtual camera 08001. The virtual parameter may be derived by calculation, or may be derived by referring to a lookup table. As the virtual camera parameters, for example, a matrix representing external parameters and a matrix representing internal parameters are used. Here, the position and orientation of the virtual camera 08001 are included in the external parameters, and the zoom value is included in the internal parameters.

  The virtual camera constraint management unit 08103 acquires and manages information for specifying a restricted area in which designation of a viewpoint based on an instruction received by the virtual camera operation unit 08101 is restricted. This information is, for example, restrictions on the position and orientation of the virtual camera 08001, the zoom value, and the like. Unlike the camera 112, the virtual camera 08001 can freely move the viewpoint to perform shooting, but cannot always generate images from all viewpoints. For example, even if the virtual camera 08001 is pointed in a direction in which an object not reflected by any camera 112 is reflected, the captured image cannot be acquired. Further, when the zoom magnification of the virtual camera 08001 is increased, the image quality is deteriorated due to the restriction of the resolution. Therefore, a zoom magnification or the like in a range that maintains a certain standard image quality may be used as the virtual camera constraint. The virtual camera constraint may be derived in advance from, for example, the arrangement of cameras. The transmission unit 06120 may reduce the amount of transmission data according to the load on the network. Due to this data amount reduction, parameters relating to a captured image change, and a range in which an image can be generated and a range in which image quality can be maintained dynamically change. The virtual camera constraint management unit 08103 may be configured to receive information indicating the method used for reducing the amount of output data from the transmission unit 06120, and dynamically update the virtual camera constraint according to the information. Thus, even if the data amount is reduced by the transmission unit 06120, it is possible to maintain the image quality of the virtual viewpoint image at a fixed reference.

  Further, the restrictions on the virtual camera are not limited to the above. In the present embodiment, the restricted area (the area that does not satisfy the virtual camera constraint) in which the specification of the viewpoint is restricted includes the operation state of the device included in the image processing system 100 and the parameters related to the image data for generating the virtual viewpoint image. It changes according to at least one of them. For example, the restricted area changes according to a parameter that is controlled so that the data amount of image data transmitted in the image processing system 100 falls within a predetermined range. The parameters include at least one of a frame rate of image data, a resolution, a quantization step, and a photographing range. For example, when the resolution of image data is reduced to reduce the amount of transmission data, the range of zoom magnification that can maintain a predetermined image quality changes. In such a case, when the virtual camera constraint management unit 08103 acquires information for specifying the restricted area that changes according to the parameter, the virtual camera operation UI 330 allows the user to specify the viewpoint within a range corresponding to the parameter change. Can be controlled to be done. The contents of the parameters are not limited to those described above. In the present embodiment, the image data of which the data amount is controlled is data generated based on a difference between a plurality of images captured by a plurality of cameras 112, but is not limited thereto. It may be itself.

  Further, for example, the restricted area changes according to the operation state of the device included in the image processing system 100. Here, the devices included in the image processing system 100 include, for example, at least one of the camera 112 and a camera adapter 120 that performs image processing on an image captured by the camera 112 to generate image data. The operating state of the apparatus includes, for example, at least one of a normal state, a failure state, a start preparation state, and a restart state of the apparatus. For example, when one of the cameras 112 is in a failure state or a restart state, it may be impossible to specify a viewpoint at a peripheral position of the camera. In such a case, when the virtual camera constraint management unit 08103 acquires information that specifies a restricted area that changes in accordance with the operation state of the device, the virtual camera operation UI 330 can operate in a range corresponding to the change in the operation state of the device. Control can be performed so that the user can specify a viewpoint. The device related to the change of the restricted area and the operation state thereof are not limited to those described above.

  The collision determination unit 08104 determines whether the virtual camera parameters derived by the virtual camera parameter derivation unit 08102 satisfy virtual camera constraints. If the constraint is not satisfied, for example, the operation input by the operator is cancelled, control is performed so that the virtual camera 08001 does not move from a position where the constraint is satisfied, or the virtual camera 08001 is returned to a position where the constraint is satisfied.

  The feedback output unit 08105 feeds back the determination result of the collision determination unit 08104 to the operator. For example, when the virtual camera restriction is no longer satisfied due to the operation of the operator, this is notified to the operator. For example, suppose that the operator operates to move the virtual camera 08001 upward, but the destination does not satisfy the virtual camera restrictions. In this case, the operator is notified that the virtual camera 08001 cannot be moved any further upward. Examples of the notification method include a sound, a message output, a change in screen color, and a method of locking the virtual camera operation unit 08101. Furthermore, the position of the virtual camera may be automatically returned to a position that satisfies the constraint, which has the effect of facilitating the operation of the operator. When the feedback is performed by image display, the feedback output unit 08105 causes the display unit to display an image based on display control according to the restricted area based on the information acquired by the virtual camera constraint management unit 08103. For example, in response to the instruction received by the virtual camera operation unit 08101, the feedback output unit 08105 causes the display unit to display an image indicating that the viewpoint corresponding to the instruction is within the restricted area. This allows the operator to recognize that the specified viewpoint is within the restricted area and may not be able to generate a desired virtual viewpoint image, and re-designates the viewpoint to a position outside the restricted area (a position satisfying the constraint). be able to. That is, in generating the virtual viewpoint image, the viewpoint can be specified within a range that changes according to the situation. Note that the content displayed on the display unit by the virtual camera operation UI 330 as a control device that performs display control according to the restricted area is not limited to this. For example, an image may be displayed in which a portion corresponding to a restricted region in a region (for example, inside a stadium) for which a viewpoint is specified is filled with a predetermined color. In the present embodiment, the display unit is an external display connected to the virtual camera operation UI 330. However, the present invention is not limited to this, and the display unit may exist inside the virtual camera operation UI 330.

  The virtual camera path management unit 08106 manages the path of the virtual camera 08001 (virtual camera path 08002) according to the operation of the operator. The virtual camera path 08002 is a sequence of information representing the position and orientation of the virtual camera 08001 for each frame. This will be described with reference to FIG. For example, virtual camera parameters are used as information indicating the position and orientation of the virtual camera 08001. For example, information for one second in setting a frame rate of 60 frames / second is a sequence of 60 virtual camera parameters. The virtual camera path management unit 08106 transmits the virtual camera parameters determined by the collision determination unit 08104 to the back-end server 270. The back-end server 270 generates a virtual viewpoint image and a virtual viewpoint sound using the received virtual camera parameters. Further, the virtual camera path management unit 08106 has a function of adding virtual camera parameters to the virtual camera path 08002 and holding the virtual camera parameters. For example, when the virtual viewpoint image and the virtual viewpoint sound for one hour are generated using the virtual camera operation UI 330, the virtual camera parameters for one hour are stored as the virtual camera path 08002. By storing this virtual camera path, the virtual viewpoint image and virtual viewpoint sound can be generated again by referring to the image information and the virtual camera path stored in the secondary storage 02460 of the database later. become. That is, another user can reuse the virtual camera path generated by the operator performing the advanced virtual camera operation and the image information stored in the secondary storage 02460. Note that a plurality of scenes corresponding to a plurality of virtual camera paths can be stored in the virtual camera management unit 08130 so that the scenes can be selected. When a plurality of virtual camera paths are stored in the virtual camera management unit 08130, meta information such as a script of a scene corresponding to each virtual camera path, an elapsed time of a game, a designated time before and after a scene, and player information is also added. Can be input and stored. The virtual camera operation UI 330 notifies the backend server 270 of these virtual camera paths as virtual camera parameters.

  By requesting the back-end server 270 for selection information for selecting a virtual camera path, the end user terminal 190 can select a virtual camera path from the scene name, the player, the elapsed game time, and the like. The back-end server 270 notifies the end user terminal 190 of a selectable virtual camera path candidate, and the end user operates the end user terminal 190 to select a desired virtual camera path from a plurality of candidates. Then, the end user terminal 190 can interactively enjoy the image distribution service by requesting the back end server 270 to generate an image according to the selected virtual camera path.

  The authoring unit 08107 provides an editing function when an operator generates a replay image. The authoring unit 08107 extracts a part of the virtual camera path 08002 held by the virtual camera path management unit 08106 as an initial value of the replay image virtual camera path 08002 in response to a user operation. As described above, the virtual camera path management unit 08106 holds meta information such as the scene name, the player, the elapsed time, and the designated time before and after the scene in association with the virtual camera path 08002. For example, a virtual camera path 08002 in which the scene name is the goal scene and the designated time before and after the scene is 10 seconds in total before and after the scene is extracted. Also, the authoring unit 08107 sets the playback speed for the edited camera path. For example, slow playback is set for the virtual camera path 08002 while the ball flies to the goal. When changing to an image from a different viewpoint, that is, when changing the virtual camera path 08002, the user operates the virtual camera 08001 again using the virtual camera operation unit 08101.

  The virtual camera image / sound output unit 08108 outputs the virtual camera image / sound received from the back-end server 270. The operator operates the virtual camera 08001 while checking the output image and sound. Note that, depending on the content of feedback by the feedback output unit 08105, the virtual camera image / audio output unit 08108 causes the display unit to display an image based on display control according to the restricted area. For example, when the position of the viewpoint specified by the operator is included in the restricted area, the virtual camera image / audio output unit 08108 generates a virtual viewpoint image with the position near the specified position and outside the restricted area as the viewpoint. May be displayed. This reduces the time and effort required for the operator to specify the viewpoint outside the restricted area.

  Next, the end user terminal 190 used by a viewer (user) will be described. FIG. 9 is a configuration diagram of the end user terminal 190.

  The end user terminal 190 on which the service application operates is, for example, a PC (Personal Computer). Note that the end user terminal 190 is not limited to a PC, and may be a smartphone, a tablet terminal, or a high-definition large display.

  The end user terminal 190 is connected via an Internet line 9001 to a back end server 270 that distributes images. For example, the end user terminal 190 (PC) is connected to a router and an Internet line 9001 via a LAN (Local Area Network) cable or a wireless LAN.

The end user terminal 190 is connected to a display 9003 for a viewer to view a virtual viewpoint image such as a sports broadcast image, and a user input device 9002 for receiving an operation such as a viewpoint change by the viewer. . For example, the display 9003 is a liquid crystal display, and is connected to a PC via a Display Port cable.
The user input device 9002 is a mouse or a keyboard, and is connected to a PC via a USB (Universal Serial Bus) cable.

  An internal function of the end user terminal 190 will be described. FIG. 10 is a functional block diagram of the end user terminal 190.

  The application management unit 10001 converts user input information input from the basic software unit 10002, which will be described later, into a backend server command of the backend server 270 and outputs the command to the basic software unit 10002. Further, the application management unit 10001 outputs to the basic software unit 10002 an image drawing instruction for drawing the image input from the basic software unit 10002 in a predetermined display area.

  The basic software unit 10002 is, for example, an OS (Operating System), and outputs user input information input from a user input unit 10004 described later to the application management unit 10001. Further, it outputs an image and a sound input from a network communication unit 10003 to be described later to the application management unit 10001, and outputs a back-end server command input from the application management unit 10001 to the network communication unit 10003. Further, the image output unit 10001 outputs the image drawing instruction input from the application management unit 10001 to the image output unit 10005.

  The network communication unit 10003 converts the backend server command input from the basic software unit 10002 into a LAN communication signal that can be communicated over a LAN cable, and outputs the signal to the backend server 270. Then, the data is passed to the basic software unit 10002 so that the image and audio data received from the back-end server 270 can be processed.

  The user input unit 10004 acquires user input information based on a keyboard input (physical keyboard or soft keyboard) or button input, or user input information input from a user input device via a USB cable, and outputs the user input information to the basic software unit 10002. I do.

  The image output unit 10005 converts an image based on an image display instruction output from the basic software unit 10002 into an image signal, and outputs the image signal to an external display or an integrated display.

  The audio output unit 10006 outputs audio data based on the audio output instruction output from the basic software unit 10002 to an external speaker or an integrated speaker.

  The terminal attribute management unit 10007 manages the display resolution of the terminal, the type of image coding codec, and the type of terminal (whether it is a smartphone or a large display).

  The service attribute management unit 10008 manages information related to a service type provided to the end user terminal 190. For example, the type of an application mounted on the end user terminal 190 and available image distribution services are managed.

  The charging management unit 10009 manages the number of receivable image distribution scenes according to the status of the user's registration and settlement to the image distribution service and the amount charged.

  Next, a workflow according to the present embodiment will be described. A workflow in a case where a plurality of cameras 112 and microphones 111 are installed in facilities such as a stadium and a concert hall to perform photographing will be described.

  FIG. 11 is a flowchart describing the overall image of the workflow. Note that the workflow processing described below is realized by the control of the controller 300 unless otherwise specified. That is, the control of the workflow is realized by the controller 300 controlling other devices (for example, the back-end server 270 and the database 250) in the image processing system 100.

  Before the processing in FIG. 11 starts, an operator (user) who installs and operates the image processing system 100 collects necessary information (prior information) before the installation and makes a plan. Further, it is assumed that the operator has installed the equipment in the target facility before the processing in FIG. 11 starts.

  In S1100, the control station 310 of the controller 300 receives a setting based on advance information from a user. Details of S1100 will be described later with reference to FIG. Next, in step S1101, each device of the image processing system 100 executes a process for confirming the operation of the system according to a command issued from the controller 300 based on an operation by a user. Details of step S1101 will be described later with reference to FIG.

  Next, in step S1102, the virtual camera operation UI 330 outputs an image and a sound before starting shooting for a game or the like. Thereby, the user can confirm the sound collected by the microphone 111 and the image captured by the camera 112 before the competition or the like. Details of step S1102 will be described later with reference to FIG.

  Then, in S1103, the control station 310 of the controller 300 causes each microphone 111 to perform sound collection and causes each camera 112 to perform shooting. The photographing in this step is assumed to include sound collection using the microphone 111, but is not limited to this, and may be only photographing an image. Details of S1103 will be described later with reference to FIGS. If the setting made in step S1101 is to be changed, or if imaging is to be ended, the process advances to step S1104. Next, in S1104, if the setting made in S1101 is changed and the shooting is continued, the process proceeds to S1105, and if the shooting is completed, the process proceeds to S1106. The determination in S1104 is typically made based on an input from the user to the controller 300. However, it is not limited to this example. In S1105, the controller 300 changes the settings made in S1101. The content of the change is typically determined based on the user input obtained in S1104. If it is necessary to stop shooting in the setting change in this step, the shooting is stopped once, and the shooting is restarted after the setting is changed. If it is not necessary to stop shooting, the setting is changed in parallel with shooting.

  In S1106, the controller 300 edits images captured by the plurality of cameras 112 and audio collected by the plurality of microphones 111. The editing is typically performed based on a user operation input via the virtual camera operation UI 330.

  Note that the processing of S1106 and S1103 may be performed in parallel. For example, when a sports competition or a concert is distributed in real time (for example, a competition image is distributed during the competition), the shooting in S1103 and the editing in S1106 are performed simultaneously. In the case where a highlight image in a sports game is distributed after the game, editing is performed after photographing ends in S1104.

  Next, details of S1100 (pre-installation processing) described above will be described with reference to FIG. First, in S1200, the control station 310 receives an input from a user regarding information (stadium information) on a facility to be photographed.

  The stadium information in this step refers to the shape of the stadium, sound, lighting, power supply, transmission environment, three-dimensional model data of the stadium, and the like. That is, the stadium information includes the stadium shape data described above. In this embodiment, the case where the facility to be photographed is a stadium is described. This is based on the assumption that an image of a sports competition to be held at the stadium is generated. However, since there are sports competitions held indoors, the facilities to be photographed are not limited to the stadium. Also, note that the event to be photographed is not limited to competitions, because virtual viewpoint images of concerts in concert halls may be generated, and images of outdoor concerts in stadiums may be generated. Keep it.

  Next, in step S1201, the control station 310 accepts an input from a user regarding device information. The device information in this step refers to information on photographing devices such as a camera, a camera platform, a lens, and a microphone, information devices such as a LAN, a PC, a server, and a cable, and information on a bogie. However, not all of this information has to be entered.

  Next, in step S1202, the control station 310 receives an input regarding the arrangement information of the camera, the camera platform, and the microphone among the photographing devices to which the device information has been input in step S1201. The arrangement information can be input by using the above-described three-dimensional model data of the stadium.

  Next, in step S1203, the control station 310 receives a user input regarding operation information of the image processing system 100. The operation information in this step indicates a shooting target, a shooting time, a camera work, a gazing point, and the like. For example, when the shooting target is an opening ceremony in which the foreground image of a player or the like in the shot image is overwhelmingly larger than the game, the image generation method can be changed to a method suitable for the situation. In addition, the gazing point and the constraint condition of the camera work may be changed according to the competition type such as the athletics competition or the soccer competition using the field. A table of setting information composed of a combination of these pieces of operation information is managed, changed, and instructed by the control station 310. This control will be described later. From S1200 to S1203, the workflow before system installation is completed.

  Next, the details of S1101 (installation processing) described above will be described with reference to FIG. First, in S1300, the control station 310 accepts a user input regarding whether or not the installed equipment is sufficient or not. The user can determine whether there is an excess or deficiency of the installed equipment by comparing the equipment information input in S1201 with the equipment to be installed and confirming whether or not there is an excess or deficiency. Next, in S1301, the control station 310 executes the installation confirmation processing of the equipment determined to be insufficient in S1300. That is, the user can install the missing equipment between S1300 and S1301, and the control station 310 confirms that the user has installed the missing equipment.

  Next, in step S1302, the control station 310 checks the system operation before adjustment to determine whether the equipment installed in step S1301 is activated and operates normally. In the process of S1302, the user may perform a system operation check, and the result of the check may be input to the control station 310 by the user.

  Here, when an error occurs in excess or deficiency of equipment or in operation, an error notification is sent to the control station 310 (S1303). The control station 310 is in a locked state where it does not proceed to the next step until the error is cleared. When the error state is released, a normal notification is made to the control station 310 (S1304), and the process proceeds to the next step. Thus, an error can be detected at an early stage. After the confirmation, the process proceeds to S1305 for the process related to the camera 112, and proceeds to S1308 for the process related to the microphone 111.

  First, the camera 112 will be described. In S1305, the control station 310 adjusts the installed camera 112. The adjustment of the camera 112 in this step refers to field angle adjustment and color adjustment, and is performed for all the installed cameras 112. The adjustment in S1305 may be performed based on a user operation, or may be realized by an automatic adjustment function.

  In the field angle adjustment, zoom, pan, tilt, and focus adjustments are performed in parallel, and the adjustment results are stored in the control station 310. Then, in the color matching, adjustment of IRIS, ISO / gain, white balance, sharpness, and shutter speed is performed simultaneously, and the adjustment results are stored in the control station 310.

  Next, in S1306, the control station 310 adjusts so that all the installed cameras are synchronized. The synchronization adjustment in S1306 may be performed based on a user operation, or may be realized by an automatic adjustment function. Further, in step S1307, the control station 310 performs camera installation calibration. More specifically, the control station 310 performs adjustment so that the coordinates of all installed cameras match the world coordinates. The detailed calibration will be described with reference to FIG. The control command of the camera 112 and the communication confirmation of the network route regarding the synchronization with the time server are also performed. Then, the process waits in the post-adjustment system operation normality confirmation process until the microphone adjustment proceeds (S1311).

  Next, a process regarding the microphone 111 will be described. First, in S1308, the control station 310 adjusts the installed microphone 111. The adjustment of the microphone 111 in this step indicates a gain adjustment, which is performed for all the installed microphones. The adjustment of the microphone 111 in S1308 may be performed based on a user operation, or may be realized by an automatic adjustment function.

  Next, in S1309, the control station 310 adjusts so that all the installed microphones are synchronized. Specifically, the synchronization clock is checked. The synchronization adjustment in S1309 may be performed based on a user operation, or may be realized by an automatic adjustment function.

  Next, in S1310, the control station 310 adjusts the position of the microphones 111 installed in the field among the installed microphones 111. The adjustment of the position of the microphone 111 in S1310 may be performed based on a user operation, or may be realized by an automatic adjustment function. The control command of the microphone 111 and the communication confirmation of the network route regarding the synchronization with the time server are also performed.

  Next, in S1311, the control station 310 performs a post-adjustment system operation check for the purpose of checking whether the cameras 112a to 112z and the microphones 111a to 111z have been correctly adjusted. The process of S1311 can be executed based on a user instruction. If both the camera 112 and the microphone 111 have confirmed the normal operation of the system after the adjustment, in S1313, a normal notification is made to the control station 310. On the other hand, if an error has occurred, an error notification is sent to the control station 310 together with the type and individual number of the camera 112 or the microphone 111 (S1312). The control station 310 issues a readjustment instruction based on the type and individual number of the device in which the error has occurred.

  Next, the details of S1102 (pre-shooting processing) described above will be described with reference to FIG. In S1400, the virtual camera operation UI 330 displays an image processed by the back-end server 270. The operator (user) of the controller 300 can check the processing result by the back-end server 270 by checking the screen of the virtual camera operation UI 330.

  Further, the operation of S1401 is performed in parallel with S1400. In step S1401, the virtual camera operation UI 330 outputs a sound processed by the back-end server 270. The operator (user) of the controller 300 can confirm the processing result by the back-end server 270 by confirming the audio output by the virtual camera operation UI 330.

  Next, in step S1402, the virtual camera operation UI 330 outputs a result in which the image and the sound processed by the back-end server 270 are combined and converted into a distribution signal. The operator (user) of the controller 300 can check the processed image and sound by the back-end server 270 by checking the output of the distribution signal by the virtual camera operation UI 330.

  Next, details of the above-described S1103 (processing at the time of shooting) will be described with reference to FIGS.

  In step S1103, the control and confirmation operations of the system are performed in the control station 310, and the operation of generating images and sounds is performed in the virtual camera operation UI 330.

  FIG. 15 illustrates the control and confirmation operations of the system, and FIG. 16 illustrates the operation of generating images and sounds. First, a description will be given with reference to FIG. In the control and confirmation operation of the system performed by the control station 310, the control and confirmation operation of the image and the sound are performed independently and simultaneously.

  First, an operation regarding an image will be described. In S1500, the virtual camera operation UI 330 displays the virtual viewpoint image generated by the back-end server 270. Next, in step S1501, the virtual camera operation UI 330 accepts an input regarding a result of confirmation by the user of the image displayed in step S1500. Then, in S1502, if it is determined that the shooting is to be ended, the process proceeds to S1508, and if it is determined that the shooting is to be continued, the process returns to S1500. That is, S 1500 and S 1501 are repeated while the photographing is continued. The control station 310 can determine whether to end or continue shooting, for example, in response to a user input.

  Next, an operation related to voice will be described. In step S <b> 1503, the virtual camera operation UI 330 accepts a user operation regarding the result of selecting the microphone 111. When the microphones 111 are selected one by one in a predetermined order, the user operation is not necessarily required. In step S1504, the virtual camera operation UI 330 reproduces the sound of the microphone 111 selected in step S1503. In step S1505, the virtual camera operation UI 330 confirms whether or not there is noise in the sound reproduced in step S1504. The determination of the presence or absence of noise in S1505 may be performed by the operator (user) of the controller 300, may be automatically determined by voice analysis processing, or both may be executed. . When the user determines the presence or absence of noise, in step S1505, the virtual camera operation UI 330 receives an input regarding the noise determination result by the user. If noise is confirmed in step S1505, in step S1506, the virtual camera operation UI 330 performs microphone gain adjustment. The adjustment of the microphone gain in S1506 may be performed based on a user operation, or may be performed automatically. If the microphone gain is adjusted based on the user operation, in step S1506, the virtual camera operation UI 330 receives a user input related to the adjustment of the microphone gain, and performs the adjustment of the microphone gain based on the user input. . Note that the selected microphone 111 may be stopped depending on the state of the noise. If it is determined in S1507 that sound collection is to be ended, the process proceeds to S1508, and if it is determined that sound collection is to be continued, the process returns to S1503. That is, the operations of S1503, S1504, S1505, and S1506 are repeated while sound collection is continued. The control station 310 can determine whether to end or continue sound collection, for example, in response to a user input. Note that the control station 310 can determine whether to end or continue sound collection, for example, in response to a user input.

  In S1508, if it is determined that the system is to be terminated, the process proceeds to S1509, and if it is determined that the system is to be continued, the process proceeds to S1500 and S1503. The determination in S1508 may be made based on a user operation. In S1509, logs acquired by the image processing system 100 are collected in the control station 310.

  Next, an operation of generating an image and a sound will be described with reference to FIG. In the operation of generating images and sounds performed by the virtual camera operation UI 330 described above, images and sounds are generated independently and in parallel.

  First, an operation regarding an image will be described. In S1600, the virtual camera operation UI 330 issues an instruction for generating a virtual viewpoint image to the back-end server 270. Then, in S1600, the back-end server 270 generates a virtual viewpoint image according to an instruction from the virtual camera operation UI 330. If it is determined in step S1601 that the image generation is to be ended, the process proceeds to step S1604. If it is determined that the image generation is to be continued, the process returns to step S1600. The determination in S1601 can be executed in response to a user operation.

  Next, an operation related to voice will be described. In step S1602, the virtual camera operation UI 330 issues an instruction for generating a virtual viewpoint sound to the back-end server 270. In step S1602, the back-end server 270 generates a virtual viewpoint sound in accordance with an instruction from the virtual camera operation UI 330. In S1603, when it is determined that the voice generation is to be ended, the process proceeds to S1604, and when it is determined that the voice generation is to be continued, the process returns to S1602. Note that the determination in S1603 may be performed in conjunction with the determination in S1601.

  Next, the workflow at the time of installation and before shooting will be described. The image processing system 100 can perform switching control between a state in which calibration is performed at the time of installation and a state in which normal imaging is performed by changing an operation mode. In some cases, it is necessary to calibrate a specific camera during shooting, and in this case, two types of operations, shooting and calibration, are compatible.

  The installation calibration process will be described with reference to the flowchart shown in FIG. In FIG. 17, a description of notification of completion of data reception and processing completion for an instruction performed between apparatuses is omitted, but some response to the instruction is returned.

  First, when the installation of the camera 112 is completed, the user instructs the control station 310 to execute the installation calibration. Then, the control station 310 instructs the front-end server 230 and the camera adapter 120 to start calibration (S04100).

  Upon receiving the calibration start instruction, the front-end server 230 determines the image data received thereafter as calibration data, and changes the control mode so that the calibration unit 02140 can process (S04102a). Further, when receiving the calibration start instruction, the camera adapter 120 shifts to a control mode in which an uncompressed frame image is handled without performing image processing such as foreground / background separation (S04102b). Further, the camera adapter 120 instructs the camera 112 to change the camera mode (S04101). The camera 112 that has received this sets the frame rate to 1 fps, for example. Alternatively, a mode in which the camera 112 transmits a still image instead of a moving image may be set (S04102c). Alternatively, a mode may be set in which the frame rate is controlled by the camera adapter 120 and the calibration image is transmitted.

  The control station 310 instructs the camera adapter 120 to acquire the zoom value and the focus value of the camera (S04103), and the camera adapter 120 transmits the zoom value and the focus value of the camera 112 to the control station 310 (S04103). S04104).

  Although only one camera adapter 120 and one camera 112 are shown in FIG. 17, control relating to the camera adapter 120 and the camera 112 is executed for all camera adapters 120 and all cameras 112 in the image processing system 100, respectively. Is done. Therefore, S04103 and S04104 are executed for the number of cameras, and when the processing of S04103 and S04104 for all the cameras 112 is completed, the control station 310 is in a state where it can receive the zoom value and the focus value for all the cameras.

  The control station 310 transmits the zoom value and the focus value for all the cameras received in S04104 to the front-end server 230 (S04105).

  Next, the control station 310 notifies the front end server 230 of the photographing pattern of the photographing for installation-time calibration (S04106).

  Here, the attribute of a pattern name (for example, pattern 1-10) for distinguishing images captured at different timings when a marker or the like serving as an image feature point is moved in the ground and captured multiple times, Is added. That is, the front-end server 230 determines that the image data for calibration received after S04106 is a photographed image in the photographing pattern received in S04106. Then, the control station 310 instructs the camera adapter 120 to perform synchronous still image shooting (S04107), and the camera adapter 120 instructs the camera 112 to perform still image shooting synchronized with all cameras (S04108). Then, the camera 112 transmits the captured image to the camera adapter 120 (S04109).

  When there are a plurality of gazing point groups, the calibration image capturing from S04106 to S04111 may be performed for each gazing point group.

  Then, the control station 310 instructs the camera adapter 120 to transmit the image instructed in S04107 to the front-end server 230 (S04110). Further, the camera adapter 120 transmits the image received in S04109 to the front-end server 230 specified as the transmission destination (S04111).

  It is assumed that the calibration image transmitted in S04111 is not subjected to image processing such as foreground / background separation, and the captured image is transmitted without compression. Therefore, when all the cameras shoot at high resolution or when the number of cameras increases, there is a possibility that all uncompressed images cannot be transmitted at the same time due to transmission band restrictions. As a result, there is a possibility that the time required for calibration in the workflow becomes longer. In this case, in the image transmission instruction of S04110, the transmission instruction of the non-compressed image according to the calibration pattern attribute is sequentially performed for each of the camera adapters 120. Further, in such a case, since it is necessary to photograph more characteristic points according to the pattern attribute of the marker, an image for calibration using a plurality of markers is photographed. In this case, from the viewpoint of load distribution, image capturing and uncompressed image transmission may be performed asynchronously. Further, the non-compressed image acquired by the image capturing for calibration is sequentially stored in the camera adapter 120 for each pattern attribute, and the non-compressed image is transmitted in parallel according to the image transmission instruction in S04110. As a result, there is an effect that the processing time of the workflow and the human error can be reduced.

  When the processing of S04111 is completed in all the cameras 112, the front-end server 230 is in a state in which the captured images of all the cameras can be received.

  As described above, when there are a plurality of shooting patterns, the processing from S04106 to S04111 is repeated for the number of patterns.

  Next, when all the calibration shootings are completed, the control station 310 instructs the front-end server 230 to perform a camera parameter estimation process (S04112).

  Upon receiving the camera parameter estimation processing instruction, the front end server 230 performs camera parameter estimation processing using the zoom values and focus values for all cameras received in S04105 and the captured images for all cameras received in S04111. (S04113). Details of the camera parameter estimation processing in S04113 will be described later. When there are a plurality of gazing points, the camera parameter estimation process of S04113 is performed for each gazing point group.

  Then, the front-end server 230 transmits the camera parameters for all cameras derived as a result of the camera parameter estimation processing in S04113 to the database 250 and stores them (S04114).

  The front-end server 230 also transmits camera parameters for all cameras to the control station 310 (S04115). The control station 310 transmits camera parameters corresponding to each camera 112 to the camera adapter 120 (S04116), and the camera adapter 120 stores the received camera parameters of the own camera 112 (S04117).

  Then, the control station 310 checks the calibration result (S04118). As a confirmation method, the derived numerical values of the camera parameters may be confirmed, the calculation process of the camera parameter estimating process in S04114 may be confirmed, or an image may be generated using the camera parameters. The image may be checked. Then, the control station 310 instructs the front end server 230 to end the calibration (S04119).

  Upon receiving the calibration end instruction, the front end server 230 changes the control mode so as to determine that the image data received thereafter is not the calibration data, contrary to the calibration start process executed in S04101 (S04120). ).

  Through the above processing, the camera parameters for all the cameras can be derived as the installation calibration processing, and the derived camera parameters can be stored in the camera adapter 120 and the database 250.

  In addition, the above-described calibration at the time of installation is performed after the camera is installed and before photographing, and it is not necessary to perform the processing again if the camera is not moved. The same processing is performed again when changing the viewpoint.

  Further, when the camera 112 has moved beyond a predetermined threshold value due to an accident such as a ball hitting during shooting, the camera 112 is shifted from a shooting state to a calibration start state, and the above-described installation calibration is performed. Is also good. In that case, the system needs to maintain the normal shooting state and notify the front-end server 230 that only the camera 112 is transmitting the calibration image, thereby setting the entire system to the calibration mode. Continuity of shooting Furthermore, in the daisy chain transmission of the present system, if an uncompressed image for calibration is sent to the transmission band of image data in normal photographing, the transmission band limit may be exceeded. In this case, the transmission priority of the non-compressed image is reduced, or the non-compressed image is divided and transmitted. Further, when the connection between the camera adapters 120 is 10 GbE or the like, the bandwidth can be secured by transmitting the uncompressed image in the opposite direction to the image data transmission of normal photographing by using the full-duplex feature. This has the effect.

  In addition, for example, when it is desired to change one point of interest among a plurality of points of interest, only the camera 112 of one point of interest group may perform the above-described installation calibration process again. In this case, during the calibration process, normal image capturing and virtual viewpoint image generation cannot be performed for the camera 112 of the target gazing point group. Therefore, the control station 310 is notified that the calibration process is being performed, and the control station 310 requests the virtual camera operation UI 330 to perform processing such as restricting the viewpoint operation. The front-end server 230 performs camera parameter estimation processing by controlling so as not to affect the processing of virtual viewpoint image generation.

The operation of the front-end server 230 in the pre-installation workflow S1200 and the installation workflow S1305 will be described with reference to the flowchart of FIG.
In S1200 of the pre-installation workflow, the control unit 02110 of the front-end server 230 receives an instruction to switch to the CAD data input mode from the control station 310, and switches to the CAD data input mode (S02210).

  The data input control unit 02120 receives the stadium CAD data (stadium shape data) from the control station 310 (S02220). The data input control unit 02120 transmits the received data to the non-imaging data file generation unit 02185 and the CAD data storage unit 02135. The CAD data storage unit 02135 stores the stadium shape data received from the data input control unit 02120 in a storage medium (S02230).

  In S1305 of the installation workflow, the control unit 02110 receives an instruction to switch to the calibration mode from the control station 310, and switches to the calibration mode (S02240).

  The data input control unit 02120 receives the calibration photographed image from the camera adapter 120, and transmits the calibration photographed image to the calibration unit 02140 (S02250).

  The calibration unit 02140 performs calibration to derive camera parameters (S02260). The calibration unit 02140 stores the derived camera parameters in the storage area, and transmits the camera parameters to the database 250 via the non-imaging data file generation unit 02185 and the DB access control unit 02190 (S02270).

  The operation of the database 250 in S1200 of the pre-installation workflow will be described with reference to the flowchart of FIG. The database 250 executes the processing of FIGS. 19 and 20 described below based on an instruction from the controller 300.

  In S1200 of the pre-installation workflow, the data input unit 02420 receives the stadium CAD data (stadium shape data) from the front-end server 230 and stores the data in the cache 02440 (S02510). The cache 02440 moves and stores the stored stadium CAD data to the primary storage 02450 (S02520).

  The operation of the database 250 in S1305 of the installation workflow will be described with reference to the flowchart of FIG.

  In S1305 of the installation workflow, the data input unit 02420 receives the camera parameters from the front-end server 230 and stores the data on the cache 02440 (S02530).

  The cache 02440 moves and stores the stored camera parameters to the primary storage 02450 (S02540). The control unit 02410 sets the number N of frames to be cached in accordance with an instruction from the control station 310 or the capacity of the cache 02440 (S02550).

  Next, the camera parameter estimation processing in the calibration unit 02140 of the front-end server 230 will be described with reference to the flowchart shown in FIG. The calibration unit 02140 executes a camera parameter estimation process based on an instruction from the control station 310. At the start of this sequence, the internal parameter map, the stadium data, the zoom value and focus value for all cameras, and the captured images for calibration for all cameras are already held by the calibration unit 02140. .

  First, the calibration unit 02140 specifies the camera 112 (S04201), specifies the corresponding zoom value and focus value, and derives internal parameter initial values from the specified zoom value and focus value using an internal parameter map (S04201). S04202). Until the derivation of the internal parameter initial value in S04202 is completed for all cameras, the processes of S04201 and S04202 are repeated (S04203).

  Next, the calibration unit 02140 specifies the camera 112 again, specifies a corresponding captured image for calibration (S04204), and detects a feature point (image feature point) in the image (S04205).

  Examples of the image feature points include a marker prepared for calibration, a pitch line previously drawn on the ground of a stadium, and an edge portion of a previously placed object (for example, a soccer goal or a player's waiting bench). And the like.

  Until the detection of image feature points in S04205 is completed for all cameras, the processing of S04205 and S04205 is repeated (S04206).

  Next, the calibration unit 02140 performs matching of the image feature points of the captured image for calibration in each camera 112 detected in S04205 (S04207). Then, it is determined whether or not the number of used feature points matched between the cameras 112 is equal to or smaller than a threshold (S04208). The threshold value of the number of used feature points used in S04208 may be set in advance, or may be automatically derived according to shooting conditions such as the number of cameras and the angle of view. Is used.

  If the number of used feature points is not equal to or smaller than the threshold in S04208, the calibration unit 02140 performs external parameter estimation processing for each camera 112 (S04209). Then, as a result of the external parameter estimation processing in S04209, it is determined whether the reprojection error is equal to or smaller than a threshold (S04210). The threshold value of the reprojection error used in S04210 may be set in advance, or may be automatically derived according to shooting conditions such as the number of cameras, and a value corresponding to the accuracy of the generated virtual viewpoint image is used. Can be

  If the reprojection error is not equal to or smaller than the threshold in the determination in S04210, the calibration unit 02140 determines that the error is large, and performs the erroneous detection of the image feature point in S04205 and the deletion processing of the erroneous matching of the image feature point in S04207 ( S04211).

  As a method of erroneous detection and erroneous matching determination in S04211, for example, the calibration unit 02140 may automatically delete a feature point having a large reprojection error, or a user may manually perform the operation while viewing the reprojection error and the image. May be deleted.

  Then, the calibration unit 02140 corrects the internal parameter with respect to the internal parameter initial value derived in S04202 (S04212).

  Then, the processing from S04208 to S04212 is repeated until the reprojection error becomes equal to or less than the threshold in S04210 within the range where the number of used feature points does not become equal to or less than the threshold in S04208.

  In the determination of S04208, the calibration unit 02140 determines that the calibration has failed if the number of used feature points is equal to or smaller than the threshold (S04213). If the calibration fails, measures such as starting over from the calibration shooting are taken. The result of the determination of success or failure is sequentially notified to the control station 310, and the control station 310 integrally manages measures such as performing a calibration process after the failure.

  If the reprojection error is equal to or smaller than the threshold in the determination of S04210, the calibration unit 02140 performs rigid transformation from the camera coordinate system to the world coordinate system for the external parameter coordinates estimated in S04209 using the stadium data ( S04214).

  The stadium data used here includes the origin of each of the X / Y / Z axes (for example, the center point of the center circle on the pitch) and the coordinate values of a plurality of feature points (for example, the intersections of pitch lines) in the stadium. , Coordinate values for performing rigid transformation are defined.

  However, when the stadium data does not exist or the accuracy of the data is low, the input of the world coordinates for performing the rigid body transformation may be manually performed, or the data for indicating the world coordinates may be the calibration unit. 02140 may be provided separately.

  In addition, since the world coordinates in the captured image for calibration are derived by performing the process of S04214, the coordinates of the feature points in the stadium recorded in the stadium data in advance are more accurately calculated using the derived results. May be updated.

  Through the above processing, camera parameters for all cameras are derived as a camera parameter estimation processing flow, and the derived camera parameters can be stored in the camera adapter 120 and the database 250.

  In a system that generates a virtual viewpoint image using images captured by a plurality of cameras, a calibration process (installation calibration) for estimating the position and orientation of each camera 112 when the camera 112 is installed is required.

  In the calibration at the time of installation, processing for obtaining camera parameters of each camera is performed. The camera parameters include internal parameters unique to the camera (such as focal length, image center, and lens distortion parameters) and external parameters (such as a rotation matrix and a position vector) representing the position and orientation of the camera. When the installation calibration processing is completed, the camera parameters of each camera are derived.

  Among the camera parameters, the internal parameters are parameters that change according to the zoom value and the focus value when the camera 112 and the lens are fixed. For this reason, in the present system, before the camera 112 is installed in the stadium, the necessary parameters for deriving the internal parameters are obtained by using the camera 112 and the lens to derive the internal parameters. Then, when the zoom value and the focus value are determined when the camera 112 is installed in the stadium, the internal parameters can be automatically derived. In this specification, this is expressed as mapping the internal parameters, and the mapping result is referred to as an internal parameter map.

  The format of the internal parameter map may be a format in which a plurality of patterns of the internal parameters corresponding to the zoom value and the focus value are recorded, or a format of an arithmetic expression capable of calculating the internal parameter value. That is, the internal parameter map may be any map as long as the internal parameters are uniquely determined according to the zoom value and the focus value.

  The parameter value obtained by the internal parameter map is used as an initial value of the internal parameter. Then, the internal parameter as a result of the camera parameter estimation processing is a value corrected in the course of the camera parameter estimation processing using an image captured for calibration after the camera 112 is installed in the stadium.

  In the present embodiment, the cameras 112 and the lenses to be installed are of the same model, and the internal parameters are the same if they have the same zoom value and the same focus value. However, the present invention is not limited to this. When there are individual differences in internal parameters even at the same zoom value and the same focus value, such as when using a plurality of types of cameras 112 and lenses, an internal parameter map is set for each model and each camera 112. You may make it hold | maintain.

  Next, a description will be given of a process of photographing by the camera 112, collecting sound by the microphone 111, and storing data photographed or collected in the database 250 via the camera adapter 120 and the front-end server 230.

  The shooting start processing sequence of the camera 112 will be described with reference to FIGS. 22A and 22B. FIGS. 22a and 22b show processing sequences having different contents, but similar results can be obtained according to any of the sequences. The camera adapter 120 selects whether to perform the processing illustrated in FIG. 22A or the processing illustrated in FIG. 22B according to the specifications of the camera 112.

  First, FIG. 22A will be described. The time server 290 performs time synchronization with, for example, the GPS 2201 and sets time managed in the time server (06801). The method is not limited to the method using the GPS 2201, and the time may be set by another method such as NTP (Network Time Protocol).

  Next, the camera adapter 120 communicates with the time server 290 using PTP (Precision Time Protocol), corrects the time managed in the camera adapter 120, and synchronizes the time with the time server 290 (06802).

  The camera adapter 120 starts providing a synchronous photographing signal such as a Genlock signal and a ternary synchronous signal and a time code signal to the camera 112 in synchronization with the photographing frame (06803). The information to be provided is not limited to the time code, but may be other information as long as it is an identifier that can identify a shooting frame.

  Next, the camera adapter 120 instructs the camera 112 to start shooting (06804).

  Upon receiving the shooting start instruction, the camera 112 performs shooting in synchronization with the Genlock signal (06805).

  Next, the camera 112 transmits the captured image including the time code signal to the camera adapter 120 (06806). Until the camera 112 stops shooting, shooting synchronized with the Genlock signal is performed.

  The camera adapter 120 performs PTP time correction processing with the time server 290 during shooting, and corrects the generation timing of the Genlock signal (06807). When the required correction amount increases, a correction according to a preset change amount may be applied.

  As described above, it is possible to realize synchronous shooting of a plurality of cameras 112 connected to a plurality of camera adapters 120 in the system.

  Next, FIG. 22B will be described. First, as in the case of FIG. 22A, time synchronization processing is performed among the camera adapter 120, the time server 290, and the GPS 2201 (06801, 06802). Next, the camera adapter 120 issues a shooting start instruction (06853). The shooting start instruction includes information for specifying the shooting period and the number of frames. The camera 112 performs shooting according to the shooting start instruction (06854).

  Next, the camera 112 transmits the captured image data to the camera adapter 120 (06855).

  The camera adapter 120 that has received the image data adds a time code to the meta information of the image data (06856).

  The camera adapter 120 performs PTP time correction processing with the time server 290 during shooting, and corrects the shooting timing of the camera 112. When the required correction amount increases, a correction according to a preset change amount may be applied. For example, a shooting start instruction is repeatedly performed at a short timing such as every frame.

  22A and 22B, the shooting start processing sequence of the camera 112 has been described. However, the microphone 111 also performs the same processing as the synchronous shooting of the camera 112 to perform synchronous sound collection. On the other hand, as the resolution of the camera image increases, the data transmission amount may exceed the network transmission band limit when the image frame of each camera 112 is transmitted. A method for reducing this risk will be described in the following embodiments.

  First, a processing sequence in which a plurality of camera adapters 120 (120a, 120b, 120c, and 120d) work together to generate three-dimensional model information in the present embodiment will be described with reference to FIG. Note that the order of processing is not limited to that shown in the figure.

  The image processing system 100 of the present embodiment includes 26 cameras 112 and a camera adapter 120. Here, two cameras 112b and 112c, and four camera adapters 120a, 120b, 120c, and The following description focuses on 120d. The camera 112b and the camera adapter 120b are connected to each other, and the camera 112c and the camera adapter 120c are connected to each other. The camera 112 connected to the camera adapter 120a and the camera adapter 120d, the microphone 111, the camera platform 113, and the external sensor 114 connected to each camera adapter 120 are omitted.

  It is also assumed that the camera adapters 120a to 120d have completed time synchronization with the time server 290 and are in a shooting state.

  The camera 112b and the camera 112c transmit the captured image (1) and the captured image (2) to the camera adapters 120b and 120c, respectively (F06301, F06302).

  The camera adapters 120b and 120c perform calibration processing on the received photographed image (1) or photographed image (2) in the calibration control unit 06133 (F06303, F06304). The calibration processing includes, for example, color correction and blur correction. Although the calibration process is performed in the present embodiment, it is not always necessary to perform the calibration process.

  Next, foreground / background separation processing is performed by the foreground / background separation unit 06131 on the captured image (1) or the captured image (2) that has been subjected to the calibration processing (F06305, F06306).

  Next, each of the separated foreground image and background image is compressed by the data compression / decompression unit 06121 (F06307, F06308). Note that the compression ratio may be changed according to the importance of each of the separated foreground image and background image. In some cases, compression need not be performed. For example, the camera adapter 120 compresses at least the background image of the foreground image and the background image and outputs the compressed image to the next camera adapter 120 so that the compression ratio of the foreground image is lower than that of the background image. When both the foreground image and the background image are compressed, lossless compression is performed on the foreground image including the important photographing target, and lossy compression is performed on the background image not including the photographing target. As a result, the amount of data transmitted to the next camera adapter 120c or the next camera adapter 120d can be efficiently reduced. For example, when shooting a field of a stadium where soccer, rugby, baseball, etc. are held, most of the image is composed of a background image, and the area of the foreground image of a player or the like is small. It should be noted here that significant savings can be made.

  Furthermore, the camera adapter 120b or the camera adapter 120c may change the frame rate of the image output to the next camera adapter 120c or the camera adapter 120d according to the importance. For example, so that the output frame rate of the background image is lower than that of the foreground image, the foreground image including the important shooting target is output at a high frame rate, and the background image not including the shooting target is output at a low frame rate. Good. As a result, the amount of data transmitted to the next camera adapter 120c or camera adapter 120d can be further reduced. Further, the compression rate and the transmission frame rate may be changed for each camera adapter 120 according to the installation location of the camera 112, the shooting location, and / or the performance of the camera 112, and the like. In addition, since the three-dimensional structure such as the seats of the stadium can be confirmed in advance using the drawing, the camera adapter 120 may transmit an image obtained by removing the seats from the background image. As a result, at the time of rendering described below, image rendering focused on the player during the match is performed by using the stadium three-dimensional structure generated in advance, and the amount of data transmitted and stored in the entire system can be reduced. This has the effect of being able to do so.

  Next, the camera adapter 120 transfers the compressed foreground image and background image to the adjacent camera adapter 120 (F06310, F06311, F06312). In the present embodiment, the foreground image and the background image are transferred at the same time, but they may be transferred individually.

  Next, the camera adapter 120b creates three-dimensional model information using the foreground image received from the camera adapter 120a and the foreground image separated by the foreground / background separation processing F06305 (F06313). Similarly, the camera adapter 120c also creates three-dimensional model information (F06314).

  Next, the camera adapter 120b transfers the foreground image and the background image received from the camera adapter 120a to the camera adapter 120c (F06315). Similarly, the camera adapter 120c transfers the foreground image and the background image to the camera adapter 120d. In the present embodiment, the foreground image and the background image are transferred at the same time, but they may be transferred individually.

  Further, the camera adapter 120c transfers the foreground image and the background image created by the camera adapter 120a and received from the camera adapter 120b to the camera adapter 120d (F06317).

  Next, each of the camera adapters 120a to 120c transfers the created three-dimensional model information to the next camera adapter 120b to 120d (F06318, F06319, F06320).

  Further, the camera adapters 120b and 120c transfer the sequentially received three-dimensional model information to the next camera adapters 120c and 120d (F06321, F06322). Further, the camera adapter 120c transfers the three-dimensional model information created by the camera adapter 120a and received from the camera adapter 120b to the camera adapter 120d (F06323).

  Finally, the foreground image, the background image, and the three-dimensional model information created by the camera adapters 120a to 120d are sequentially transmitted between the network-connected camera adapters 120 and transmitted to the front-end server 230.

  In the sequence diagram, the description of the calibration processing, the foreground / background separation processing, the compression processing, and the three-dimensional model information creation processing of the camera adapters 120a and 120d is omitted. However, actually, the camera adapter 120a and the camera adapter 120d also perform the same processing as the camera adapter 120b and the camera adapter 120c, and create the foreground image, the background image, and the three-dimensional model information. Although the data transfer sequence between the four camera adapters 120 has been described here, the same processing is performed even if the number of camera adapters 120 increases.

  As described above, among the plurality of camera adapters 120, the camera adapters 120 other than the last camera adapter 120 in a predetermined order extract a predetermined area from the image captured by the corresponding camera 112. Then, image data based on the extraction result is output to the next camera adapter 120 in the above-described predetermined order. On the other hand, the last camera adapter 120 in the above-described predetermined order outputs image data based on the extraction result to the image computing server 200. That is, the plurality of camera adapters 120 are connected in a daisy chain, and image data based on the result of each camera adapter 120 extracting a predetermined area from the captured image is input to the image computing server 200 by the predetermined camera adapter 120. You. By using such a data transmission method, it is possible to suppress a change in the processing load on the image computing server 200 and a change in the network transmission load when the number of the sensor systems 110 in the image processing system 100 changes. . The image data output by the camera adapter 120 is generated using the image data based on the above extraction result and the image data based on the extraction result of the predetermined region by the previous camera adapter 120 in a predetermined order. It may be data. For example, each camera adapter 120 outputs image data based on the difference between the extraction result by itself and the extraction result by the previous camera adapter 120, so that the amount of transmission data in the system can be reduced. The last camera adapter 120 in the above order acquires, from the other camera adapter 120, extracted image data based on image data of a predetermined area extracted by the other camera adapter 120 from an image captured by the other camera 112. Then, it outputs image data corresponding to the extraction result of the predetermined area extracted by itself and the extracted image data acquired from another camera adapter 120 to the image computing server 200 for generating a virtual viewpoint image. .

  Further, the camera adapter 120 divides an image captured by the camera 112 into a foreground portion and a background portion, and changes a compression rate and a frame rate to be transmitted, for example, according to their importance. As a result, the transmission amount can be reduced as compared with the case where all the data captured by the camera 112 is transmitted to the front-end server 230. In addition, each camera adapter 120 sequentially creates three-dimensional model information necessary for generating a three-dimensional model. As a result, all data is gathered in the front-end server 230, and the processing load on the server can be reduced as compared with the case where all three-dimensional model generation processing is performed by the server. Can be made possible.

  Next, a flow of processing for generating a foreground image and a background image and transferring the generated foreground image and background image to the next camera adapter 120 in the sequential generation of three-dimensional model information in the camera adapter 120 will be described with reference to FIG.

  The camera adapter 120 acquires a captured image from the camera 112 connected thereto (06501).

  Next, a process of separating the obtained captured image into a foreground image and a background image is performed (06502). Note that the foreground image in the present embodiment is an image determined based on a detection result of a predetermined object in a captured image acquired from the camera 112. The predetermined object is, for example, a person. However, the object may be a specific person (such as a player, a manager, and / or a referee), or may be an object with a predetermined image pattern, such as a ball or a goal. Further, a moving object may be detected as an object.

  Next, compression processing of the separated foreground image and background image is performed. Lossless compression is performed on the foreground image, and the foreground image maintains high image quality. Lossless compression is performed on the background image, and the amount of transmission data is reduced (06503).

  Next, the camera adapter 120 transfers the compressed foreground image and background image to the next camera adapter 120 (06504). Note that the background image may not be transferred every frame, but may be transferred by thinning out transfer frames. For example, when the captured image is 60 fps, the foreground image is transmitted every frame, but the background image is transmitted only one frame out of 60 frames per second. Thereby, there is a specific effect that the amount of transmission data can be reduced.

  The camera adapter 120 may add meta information when transferring the foreground image and the background image to the next camera adapter 120. For example, the identifier of the camera adapter 120 or the camera 112, the position (xy coordinates) of the foreground image in the frame, the data size, the frame number, the shooting time, and the like are given as meta information. Also, gazing point group information for identifying the gazing point, data type information for identifying the foreground image and the background image, and the like may be added. However, the content of the data to be provided is not limited to these, and other data may be provided.

  Note that when the camera adapter 120 transmits data through the daisy chain, the transmission processing load on the camera adapter 120 is reduced by selectively processing only the image captured by the camera 112 having a high correlation with the camera 112 connected to the camera adapter 120. Can be reduced. In daisy chain transmission, robustness can be ensured by configuring the system so that data transmission between the camera adapters 120 does not stop even if a failure occurs in any of the camera adapters 120.

  Next, the flow of processing when data is received from the adjacent camera adapter 120 in the three-dimensional model information generation processing flow in the camera adapter 120 will be described using FIG.

  First, the camera adapter 120 receives data from the adjacent camera adapter 120 (S06601). The camera adapter 120 determines whether or not its own transfer mode is the bypass control mode (S06602). The bypass control will be described with reference to FIG.

  In the case of the bypass control mode, the camera adapter 120 transfers data to the next camera adapter 120 (S06611). If the mode is not the bypass control mode, the received data packet is analyzed (S06603).

  As a result of analyzing the packet, when the camera adapter 120 determines that the packet is a bypass transmission control target packet (Yes in S06604), the camera adapter 120 transfers the data to the next camera adapter 120 (S06610). The bypass transmission control target packet is, for example, image data not used for generating three-dimensional model information, a control message described later, or a message related to time correction. The bypass transmission control will be described with reference to FIG.

  If the camera adapter 120 determines that the data is not the bypass transmission control target, the camera adapter 120 determines the data type (S06605), and performs a process according to the data type.

  If the data type is a control message packet addressed to the camera adapter 120 from the control station 310, the control message is analyzed and processing is performed based on the analysis result (S06606). The same applies when the source of the control message is not the control station 310 but another node. The same applies not only when the packet is addressed to the camera adapter 120 itself, but also when the packet is addressed to the point of interest group to which the camera adapter 120 belongs. Examples of processing performed by the camera adapter 120 include control of the microphone 111, camera 112, and camera platform 113 connected to the camera adapter 120, and control of the camera adapter 120 itself. The camera adapter 120 returns a control result to the transmission source or the designated node according to the content of the control message. If the packet is a control message addressed to the group, the control message is transferred to the next camera adapter 120.

  Next, when the data type relates to time correction, the camera adapter 120 performs time correction processing (S06607). For example, its own time is corrected based on the PTP processing with the time server 290. Then, the word clock supplied to the microphone 111 and the camera 112 is corrected based on the corrected time. Note that if the word clock timing is changed at one time when the time correction amount is large, the sound and image quality will be affected. Therefore, the time may be gradually corrected based on a preset change amount. Further, the camera adapter 120 transfers the created three-dimensional model information and the foreground image used for creating the three-dimensional model information to the next camera adapter 120 for transmission to the front-end server 230.

  Next, when the data type is the foreground image or the background image, the camera adapter 120 performs three-dimensional model information creation processing (S06608).

  Next, control according to the point of interest group will be described. FIG. 26 is a diagram for explaining a point of interest group. Each camera 112 is installed so that the optical axis faces a specific point of regard 06302. The cameras 112 classified into the same gazing point group 06301 are installed so as to face the same gazing point 06302.

  FIG. 26 shows an example in which two gazing points 06302, a gazing point A (06302A) and a gazing point B (06302B), are set and nine cameras (112a to 112i) are installed. The four cameras (112a, 112c, 112e, and 112g) face the same gazing point A (06302A) and belong to the gazing point group A (06301A). The remaining five cameras (112b, 112d, 112f, 112h, and 112i) face the same gazing point B (06302B) and belong to the gazing point group B (06301B).

  Here, the closest set (the number of connection hops) of the cameras 112 belonging to the same gazing point group 06301 is expressed as being logically adjacent. For example, the camera 112a and the camera 112b are physically adjacent to each other, but are not logically adjacent to each other because they belong to different gazing point groups 06301. Logically adjacent to camera 112a is camera 112c. On the other hand, the camera 112h and the camera 112i are not only physically adjacent but also logically adjacent.

  Different processing is performed by the camera adapter 120 depending on whether or not the physically adjacent cameras 112 are logically adjacent. Hereinafter, specific processing will be described.

  The bypass transmission control will be described with reference to FIG. The bypass transmission control is a function in which transmission data is bypassed according to the point of interest group to which each camera adapter 120 belongs. The description of the functional units configuring the external device control unit 06140, each image processing unit 06130, the transmission unit 06120, and the network adapter 06110 is omitted.

  In the image processing system 100, the setting of the number of camera adapters 120 and which camera adapters 120 belong to which point of interest group can be changed. In FIG. 27, it is assumed that the camera adapter 120g, the camera adapter 120h, and the camera adapter 120n belong to the gazing point group A, and the camera adapter 120i belongs to the gazing point group B.

  A route 06450 indicates a transmission route of the foreground image created by the camera adapter 120g, and the foreground image is finally transmitted to the front-end server 230. In the figure, the description of the background image, the three-dimensional model information, the control message, and the foreground image created by the camera adapter 120h, the camera adapter 120i, and the camera adapter 120n is omitted.

  The camera adapter 120h receives the foreground image created by the camera adapter 120g via the network adapter 06110h, and determines the routing destination by the transmission unit 06120h. When the transmission unit 06120h determines that the camera adapter 120g that created the received foreground image belongs to the same gazing point group (here, group A), the transmission unit 06120h transfers the received foreground image to the image processing unit 06130h. When the image processing unit 06130h generates the three-dimensional model information based on the foreground image created and transmitted by the camera adapter 120g, the foreground image of the camera adapter 120g is transferred to the next camera adapter 120i.

  Next, the camera adapter 120i receives the foreground image created by the camera adapter 120g from the camera adapter 120h. If the transmission unit 06120i of the camera adapter 120i determines that the gazing point group to which the camera adapter 120g belongs is different from the camera adapter 120g, the transmission unit 06120i transfers the camera adapter 120g to the next camera adapter 120 without transferring it to the image processing unit 06130i.

  Next, the camera adapter 120n receives the foreground image created by the camera adapter 120g via the network adapter 06110n, and determines a routing destination by the transmission unit 06120n. The transmission unit 06120n determines that the camera adapter 120n is in the same gazing point group as the camera adapter 120g. However, when the image processing unit 06130n determines that the foreground image of the camera adapter 120g is not a foreground image necessary for generating three-dimensional model information, the foreground image is directly transferred to the next camera adapter 120 via the daisy chain network. Is done.

  As described above, the transmission unit 06120 of each camera adapter 120 determines whether the received data is data necessary for creating three-dimensional model information by image processing in the image processing unit 06130. If it is determined that the data is not necessary for image processing, that is, the data has a low correlation with the camera adapter 120 itself, the data is transmitted to the next camera adapter 120 without transferring the data to the image processing unit 06130. That is, in data transmission via the daisy chain 170, a process of selecting necessary data in each camera adapter 120 and sequentially generating three-dimensional model information is performed. As a result, the processing load and processing time related to data transfer from when data is received to when data is transferred in the camera adapter 120 can be reduced.

  Next, the bypass control of the camera adapter 120b will be described in more detail with reference to FIG. It should be noted that the description of the functional units constituting the external device control unit 06140, each image processing unit 06130, the transmission unit 06120, and the network adapter 06110 is omitted.

  The bypass control is a function in which the camera adapter 120b transfers data received from the camera adapter 120c to the next camera adapter 120a without performing routing control by the data routing processing unit 06122 of the transmission unit 06120.

  For example, the camera adapter 120b activates bypass control for the network adapter 06110 when the state of the camera 112b is stopping photographing, performing calibration, or performing error processing. Also, for example, when an operation failure of the transmission unit 06120 or the image processing unit 06130 occurs, the bypass control is activated. Alternatively, the network adapter 06110 may detect the state of the transmission unit 06120 and actively transition to the bypass control mode. A sub CPU for detecting that the transmission unit 06120 or the image processing unit 06130 is in an error state or a halt state is provided in the camera adapter 120b, and when the sub CPU detects an error, the network adapter 06110 is set to bypass control. Processing may be added. Thus, there is an effect that the fault state and the bypass control of each functional block can be controlled independently.

  Also, the camera adapter 120 may transition from the bypass control mode to the normal communication mode when the state of the camera 112 transitions from the calibration state to the shooting state, or when the transmission unit 06120 or the like recovers from malfunction. Good.

  With this bypass control function, the camera adapter 120 can perform high-speed data transfer, and can transfer data to the next camera adapter 120a even when an unexpected failure or the like cannot be performed and a determination relating to data routing cannot be made. .

  In this system, the foreground image, the background image, and the three-dimensional model information are transmitted between the plurality of camera adapters 120 connected in a daisy chain and input to the front-end server 230. Here, when an event in which the foreground area is extremely large in the captured image, for example, when an opening ceremony in which all players gather is taken, the amount of data of the foreground image transmitted is larger than that in a case where a normal competition is taken. Become huge. Therefore, a method for controlling the amount of data transmitted in the daisy chain so as not to exceed the transmission band will be described below.

  A flow of a process in which the transmission unit 06120 outputs data in the camera adapter 120 will be described with reference to FIGS. 29 and 30. FIG. 29 shows the data flow between the camera adapters 120a, 120b and 120c. The camera adapter 120a and the camera adapter 120b, and the camera adapter 120b and the camera adapter 120c are respectively connected. The camera adapter 120b is connected to the camera 112b, and the camera adapter 120c is connected to the front-end server 230. A data output processing flow of the transmission unit 06120 of the camera adapter 120b will be described.

  Image data 06720 is input from the camera 112b to the transmission unit 06120 of the camera adapter 120b, and input data 06721 and 06722 that have been subjected to image processing are input from the camera adapter 120a. The transmission unit 06120 performs processing such as output to the image processing unit 06130, compression, setting of a frame rate, and packetization on the input data, and outputs the data to the network adapter 06110. .

  Next, an output processing flow by the transmission unit 06120 will be described using FIG. The transmission unit 06120 executes a step of acquiring the data amount of the image processing result for each of the input data 06721 and 06720 from the image processing unit 06130 (S06701).

  Next, a step of acquiring the data amount of the input data 06722 from the camera adapter 120a (S06702) is executed. Next, a step of deriving the output data amount to the camera adapter 120c according to the data type of the input data (S06703) is executed.

  Next, the transmission unit 06120 compares the output data amount with a predetermined transmission band restriction amount, and confirms the transmission possibility. More specifically, it is determined whether or not the amount of data to be output to the network adapter 06110 exceeds a predetermined output data amount threshold (S06704). The threshold may be provided for each data type (here, foreground image, background image, whole-view frame data, three-dimensional model information, and the like). The amount of data to be output is derived based on the result of compression when the transmission unit 06120 compresses data. It is desirable that the threshold value of the output data amount is set in consideration of overhead such as header information and error correction information when packetizing.

  If the transmission unit 06120 determines that the output data amount does not exceed the threshold, the transmission unit 06120 performs a normal transfer of outputting the input data to the network adapter 06110 (S06712). If it is determined that the output data amount has exceeded the threshold (Yes in S6704), if the data input to the transmission unit 06120 is image data, a policy for when the output data amount is over is acquired (S06705). Then, based on the acquired policy, at least one of a plurality of processes (S06707-S06711) described below is selected and executed (S06706). Note that the transmission unit 06120 may perform normal transfer of data relating to time correction and data relating to a control message other than image data. Also, the message may be dropped according to the type and priority of the message. By reducing the amount of output data, it is possible to suppress overflow of data transfer.

  As one of the processes executed by the transmission unit 06120, the frame rate of the image data is reduced and output to the network adapter 06110 (S06707). The data amount is reduced by thinning out and transmitting the frames. However, when following a fast-moving object, there is a possibility that the image quality is inferior to the case of outputting at a high frame rate. Therefore, the applicability of this method is determined according to the target shooting scene.

  As another process, the transmission unit 06120 reduces the resolution of the image data and outputs the image data to the network adapter 06110 (S06708). Since this process affects the image quality of the output image, a policy is set according to the type of the end user terminal. For example, when outputting to a smartphone, a policy is set for adaptive resolution conversion such as reducing the data amount by greatly reducing the resolution, and when outputting to a high-resolution display or the like, reducing the resolution to a small value.

  As another process, the transmission unit 06120 increases the compression ratio of the image data and outputs the image data to the network adapter 06110 (S06709). Here, for the input image data, the data amount is reduced in accordance with a restoration performance request such as lossless compression or lossy compression, that is, a request for image quality.

  As another process, the transmission unit 06120 stops outputting the imaging data 06720 from the image processing unit 06130 (S06710). Here, the output of the image data subjected to the image processing is stopped to reduce the data amount. When a sufficient number of cameras 112 are provided, all the cameras 112 of the same gazing point group may not be essential in generating the virtual viewpoint image. For example, this control is applied when it is previously known that a blind spot does not occur even if the number of cameras 112 is reduced in photographing the entire stadium field. That is, a transmission band can be secured by selecting a camera that does not transmit image data, on condition that no image failure occurs in a subsequent process.

  As another process, the transmission unit 06120 stops the output of the input data 06721 from the image processing unit 06130, or stops the output of only a part of the image of the camera adapter 120 (S06711). In addition to the above, when three-dimensional model information can be generated using an input image from another camera adapter 120, output of a foreground image or a background image from another camera adapter 120 is stopped, The data amount may be reduced by controlling the output of only the model information.

  The method used to reduce the amount of output data is notified to the back-end server 270, the virtual camera operation UI 330, and the control station 310 via the front-end server 230 at the subsequent stage (S06713). In the present embodiment, the flow branches so that any one of the control processes such as the frame rate, the resolution, the compression ratio, and the data stop is performed according to the policy. However, the present invention is not limited to this. It is specified that a further data amount reduction effect can be obtained by executing a plurality of these controls in combination. Further, in S06713, a notification of the present control processing is performed. With this notification, in the virtual camera operation UI 330, for example, if a sufficient resolution cannot be obtained in terms of image quality as a result of increasing the compression ratio, a restriction can be imposed on the zoom operation. Further, even after the transmission band restriction amount over process, it is sequentially shown that the output data amount is exceeded, and when the data amount is stabilized, it is shown here that the transmission processing policy can be returned to the original set value.

  As described above, in order to solve the problem of exceeding the transmission band of the daisy chain, there is an effect that transmission satisfying the transmission band restriction can be realized by performing the transmission control processing according to the state.

  Next, the operation of the front-end server 230 in S1500 and S1600 of the workflow at the time of shooting will be described with reference to the flowchart of FIG.

  The control unit 02110 receives the instruction to switch to the shooting mode from the control station 310, and switches to the shooting mode (S02300). When the photographing is started, the data input control unit 02120 starts receiving the photographing data from the camera adapter 120 (S02310).

  The photographing data is buffered by the data synchronizing unit 02130 until all the photographing data necessary for file creation are prepared (S02320). Although not explicitly shown in the flowchart, here, it is determined whether or not the time information given to the shooting data matches, and whether or not a predetermined number of cameras are satisfied. Depending on the state of the camera 112, image data may not be sent due to calibration or error processing. In this case, the fact that the image of the predetermined camera number is missing is notified in the subsequent database 250 transfer (S2370). Here, in order to determine whether a predetermined number of cameras are satisfied, there is a method of waiting for arrival of shooting data for a predetermined time. However, in the present embodiment and the like, in order to suppress a delay in a series of processing of the system, when each camera adapter 120 transmits data through a daisy chain, information indicating presence / absence of image data corresponding to each camera number is added. . This allows the control unit 02110 of the front-end server 230 to make an immediate determination. It should be noted here that the effect of eliminating the need to set the waiting time for the photographic data is obtained.

  After the data necessary for file creation is buffered by the data synchronization unit 02130, development processing of RAW image data, lens distortion correction, and matching of colors and luminance values between images captured by each camera of the foreground image and the background image Various conversion processes are performed (S02330).

  When the data buffered by the data synchronizer 02130 includes a background image, a background image combining process (S02340) is performed, and when the data does not include a background image, a three-dimensional model combining process (S02350) is performed (S02335). ).

  In step S02330, the image combining unit 02170 acquires the background image processed by the image processing unit 02150. Then, in S02230, the background images are joined in accordance with the coordinates of the stadium shape data stored in the CAD data storage unit 02135, and the combined background image is sent to the photographing data file generation unit (S02340).

  The three-dimensional model coupling unit 02160 that has acquired the three-dimensional model from the data synchronization unit 02130 generates a three-dimensional model of the foreground image using the three-dimensional model data and the camera parameters (S02350).

  The photographing data file generation unit 02180 that has received the photographing data created by the processing up to S02350 forms the photographing data according to the file format and packs the photographing data. Thereafter, the created file is sent to the DB access control unit 02190 (S02360). The DB access control unit 02190 transmits the image data file received from the image data file generation unit 02180 in S02360 to the database 250 (S02370).

  Next, with respect to the operation of the database 250 in the generation of the virtual viewpoint image in S1500 and S1600 of the shooting workflow, particularly the data writing operation will be described with reference to the flowchart of FIG.

  The photographing data is input from the front-end server 230 to the data input unit 02420 of the database 250. The data input unit 02420 extracts time information or time code information associated with the input photographing data as meta information, and detects that the input photographing data is photographing data at time t1 (S2810).

  The data input unit 02420 sends the input photographing data at the time t1 to the cache 02440, and the cache 02440 caches the photographing data at the time t1 (S02820).

  The data input unit 02420 determines whether or not the photographing data N frames before the time t1, ie, the photographing data at the time t1-N, is cached. If cached, S02830 is completed. If not cached, the process ends. (S02825). Note that N is variable depending on the frame rate. T1-N described here may be a time that is N times the frame unit time from t1 or a time code that is N frames before the frame at time t1.

  When the cache 02440 caches the image data at time t1, the cached image data at time t1-N is transferred to the primary storage, and the primary storage 02450 records the image data at time t1-N sent from the cache 02440. (S02830). Thus, the frames before the predetermined time are sequentially stored in the primary storage according to the capacity limitation of the cache that can be accessed at high speed. This can be realized, for example, by making the cache 02440 a ring buffer structure.

  Subsequently, with respect to the operation of the database 250 in the generation of the virtual viewpoint image in S1500 and S1600 of the shooting workflow, particularly, the operation of reading data will be described with reference to the flowchart of FIG.

  The back-end server 270 requests the data output unit 02430 for data whose time code corresponds to the time t (S02810). The data output unit 02430 determines whether the data at the time t is stored in the cache 02440 or the primary storage 02450, and determines from which data to read (S02820). For example, similarly to FIG. 32 described above, when the time t1 is the time when the photographing data is input to the data input unit 02420, if the time t is a time before the time t1-N, the data from the primary storage is deleted. It is read (S02830). If the time t is a time between the times t1-N and t1, it is read from the cache (S02840). If the time t is later than the time t1, the data output unit 02430 notifies the back-end server 270 of an error (S02850).

  Next, regarding the processing flow of the image processing unit 06130 of the camera adapter 120, each flowchart of FIGS. 35 (a), 35 (b), 35 (c), 35 (d), and 35 (e) will be described. It will be described using FIG.

  Prior to the processing in FIG. 35A, the calibration control unit 06133 performs color correction processing on the input image to suppress color variation for each camera and reduces image blurring caused by camera vibration. Correction processing (electronic image stabilization processing) for stabilizing an image. In the color correction processing, processing such as adding an offset value to the pixel value of the input image is performed based on the parameters received from the front end server 230. In the blur correction process, the blur amount of an image is estimated based on output data from a sensor such as an acceleration sensor or a gyro sensor built in the camera. Then, the image position is shifted with respect to the input image or the image is rotated based on the estimated blur amount, thereby suppressing the blur between the frame images. It should be noted that other methods may be used as a blur correction method. For example, a method based on image processing for estimating and correcting the moving amount of an image by comparing a plurality of temporally consecutive frame images, a method realized inside a camera such as a lens shift method and a sensor shift method, and the like. May be.

  The background update unit 05003 performs a process of updating the background image 05002 using the input image and the background image stored in the memory. An example of the background image is shown in FIG. Update processing is performed for each pixel. The processing flow is shown in FIG.

  First, in S05001, the background update unit 05003 derives a difference between each pixel of the input image and a pixel at a corresponding position in the background image. Next, in S050002, it is determined whether or not the difference is smaller than a predetermined threshold K. If the difference is smaller than K, it is determined that the pixel is the background (YES in S5002). Next, in S05003, the background update unit 05003 derives a value obtained by mixing the pixel value of the input image and the pixel value of the background image at a fixed ratio. In step S05004, the pixel value in the background image is updated with the derived value.

  On the other hand, FIG. 34B shows an example in which a person is shown in FIG. 34A which is a background image. In such a case, paying attention to the pixel where the person is located, the difference between the pixel values with respect to the background becomes large, and the difference becomes K or more in S050002. In this case, since the change in the pixel value is large, it is determined that some object other than the background is reflected, and the background image 05002 is not updated (NO in S05002). Various other methods can be considered for the background update process.

  Next, the background extraction unit 05004 reads a part of the background image 05002 and transmits it to the transmission unit 06120. When a game such as a soccer game is shot at a stadium or the like, when a plurality of cameras 112 are arranged so that the entire field can be shot without blind spots, the feature is that most of the background information is overlapped between the cameras 112. Since the background information is enormous, it is possible to reduce the amount of transmission by deleting and transmitting the duplicated portion from the viewpoint of transmission band restrictions. FIG. 35D shows the flow of the processing. In step S05010, the background clipping unit 05004 sets a central portion of the background image, for example, as a partial region 3401 surrounded by a dotted line shown in FIG. That is, this partial area 3401 is a background area in which the own camera 112 is in charge of transmission, and the other background areas are in charge of transmission by another camera 112. In step S05011, the background extraction unit 05004 reads the set partial region 3401 of the background image. Then, in S05012, it outputs to transmission section 06120. The output background images are collected by the image computing server 200 and used as a texture of the background model. The position where the background image 05002 is cut out in each camera adapter 120 is set according to a predetermined parameter value so that the texture information for the background model is not insufficient. Usually, in order to further reduce the amount of transmission data, the area to be cut out is set to be a necessary minimum. This has the effect of reducing the amount of transmission of enormous background information, and can provide a system that can cope with higher resolution.

  Next, the foreground separation unit 05001 performs a process of detecting a foreground area (an object such as a person). FIG. 35B shows the flow of the foreground area detection process executed for each pixel. For the foreground detection, a method using background difference information is used. First, in S05005, the foreground separation unit 05001 derives a difference between each pixel of the newly input image and a pixel at a corresponding position in the background image 05002. Then, in S05006, it is determined whether or not the difference is larger than the threshold L. Here, assuming that the newly input image is, for example, as shown in FIG. 34B with respect to the background image 05002 shown in FIG. In, the difference becomes large. If the difference is larger than the threshold L, the pixel is set as the foreground in S05007. In the foreground detection method using the background difference information, various devices for detecting the foreground with higher accuracy have been considered. There are various other methods for foreground detection, such as methods using feature values and machine learning.

  After executing the processing described above with reference to FIG. 35B for each pixel of the input image, the foreground separation unit 05001 performs processing of determining and outputting the foreground area as a block. The processing flow is shown in FIG. In S05008, a foreground area in which a plurality of pixels are connected is set as one foreground image with respect to the image in which the foreground area is detected. As a process for detecting a region where pixels are connected, for example, a region growing method is used. Since the region growing method is a known algorithm, a detailed description is omitted. After the foreground areas are grouped as foreground images in S05008, the foreground images are sequentially read out in S05009 and output to the transmission unit 06120.

  Next, the three-dimensional model information generation unit 06132 generates three-dimensional model information using the foreground image. When the camera adapter receives a foreground image from an adjacent camera, the foreground image is input to the other camera foreground receiving unit 05006 via the transmission unit 06120. FIG. 35E shows a flow of a process executed by the three-dimensional model processing unit 05005 when a foreground image is input. Here, when the image computing server 200 collects image data captured by the camera 112, starts image processing, and generates a virtual viewpoint image, the amount of calculation may be large and the time required for image generation may be long. In particular, there is a possibility that the amount of calculation in generating the three-dimensional model is significantly increased. 35 (e), a method for sequentially generating three-dimensional model information while transmitting data by connecting the camera adapters 120 in a daisy chain in order to reduce the processing amount in the image computing server 200 will be described.

  First, in S05013, the three-dimensional model information generation unit 06132 receives a foreground image captured by another camera 112. Next, in 05014, the three-dimensional model information generation unit 06132 confirms whether the camera 112 that has captured the received foreground image belongs to the same gazing point group as the own camera 112 and is an adjacent camera. If S05014 is YES, the process proceeds to S05015. In the case of NO, it is determined that there is no correlation with the foreground image of the other camera 112, and the processing ends without performing the processing. In S05014, it is confirmed whether or not the camera is an adjacent camera. However, the method of determining the correlation between the cameras 112 is not limited to this. For example, a similar method may be adopted in which the three-dimensional model information generation unit 06132 previously obtains and sets the camera number of the camera 112 having a correlation, and acquires and processes the image data only when the image data of the camera 112 is transmitted. The effect is obtained.

  Next, in S05015, the three-dimensional model information generation unit 06132 derives depth information of the foreground image. Specifically, first, the foreground image received from the foreground separation unit 05001 is associated with the foreground image of another camera 112, and then, based on the coordinate values of the associated pixels and the camera parameters, each foreground image is The depth information of each pixel on the image is derived. Here, as a method of associating images, for example, a block matching method is used. Since the block matching method is a well-known method, a detailed description will be omitted. In addition, there are various other methods for associating the characteristic points to improve performance by combining feature point detection, feature amount calculation, matching processing, and the like, and any method may be used.

  Next, in S05016, the three-dimensional model information generation unit 06132 derives three-dimensional model information of the foreground image. Specifically, for each pixel of the foreground image, the world coordinate value of the pixel is derived based on the depth information derived in S05015 and the camera parameters stored in the camera parameter receiving unit 05007. Then, one point data of a three-dimensional model configured as a point group is set with the world coordinate values and the pixel values as a set. Through the above processing, part of the three-dimensional model obtained from the foreground image received from the foreground separation unit 05001 and part of the point group of the three-dimensional model obtained from the foreground image of the other camera 112 Information. In step S05017, the three-dimensional model information generation unit 06132 adds the camera number and the frame number to the obtained three-dimensional model information as meta information (the meta information may be, for example, a time code or an absolute time). Output to

  Accordingly, even when the camera adapters 120 are connected by a daisy chain and a plurality of gazing points are set, image processing is performed according to the correlation between the cameras 112 while transmitting data by the daisy chain, and the three-dimensional model Information can be generated sequentially. As a result, there is an effect that the processing is speeded up.

  In the present embodiment, each process described above is executed by hardware such as an FPGA or an ASIC mounted on the camera adapter 120, but may be executed by software using a CPU, a GPU, a DSP, or the like. Good. In this embodiment, the three-dimensional model information is generated in the camera adapter 120. However, the image computing server 200 that collects all the foreground images from the cameras 112 may generate the three-dimensional model information.

  Next, a process in which the back-end server 270 generates a live image and a replay image based on the data stored in the database 250 and displays the generated image on the end user terminal 190 will be described. Note that the back-end server 270 of the present embodiment generates virtual viewpoint content as a live image and a replay image. In the present embodiment, the virtual viewpoint content is content generated using images captured by a plurality of cameras 112 as a plurality of viewpoint images. That is, the back-end server 270 generates virtual viewpoint content based on, for example, viewpoint information specified based on a user operation. Further, in the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content is mainly described, but audio data may not necessarily be included.

  When the user operates the virtual camera operation UI 330 to specify a viewpoint, there is no image captured by the camera 112 for generating an image corresponding to the specified viewpoint position (the position of the virtual camera), or the resolution is not sufficient. Or the image quality is low. At this time, if it cannot be determined until the image generation stage that the conditions for providing the image to the user cannot be satisfied, the operability of the operator may be impaired. Hereinafter, a method for reducing this fear will be described.

  FIG. 36 shows a processing flow of the virtual camera operation UI 330, the back-end server 270, and the database 250 from when the operator (user) operates the input device until the virtual viewpoint image is displayed.

First, the operator operates the input device to operate the virtual camera (S03300).
As the input device, for example, a joystick, a jog dial, a touch panel, a keyboard, a mouse, and the like are used.

In the virtual camera operation UI 330, virtual camera parameters representing the input position and orientation of the virtual camera are derived (S03301).
The virtual camera parameters include external parameters indicating the position and orientation of the virtual camera, and internal parameters indicating the zoom magnification of the virtual camera.
The virtual camera operation UI 330 transmits the derived virtual camera parameters to the back-end server 270.

  Upon receiving the virtual camera parameters, the back-end server 270 requests the database 250 for a foreground three-dimensional model group (S03303). The database 250 transmits the foreground three-dimensional model group including the foreground object position information to the back-end server 270 in response to the request (S03304).

  The back-end server 270 geometrically derives a foreground object group that enters the field of view of the virtual camera based on the virtual camera parameters and the position information of the foreground object included in the foreground three-dimensional model (S03305).

The back-end server 270 requests the database 250 for the derived foreground image, foreground three-dimensional model, background image, and audio data group of the foreground object group (S03306).
The database 250 transmits the data to the back-end server 270 in response to the request (S03307).

  The back-end server 270 generates a foreground image and a background image of a virtual viewpoint from the received foreground image, foreground three-dimensional model, and background image, and combines them to generate a full-view image of the virtual viewpoint.

  Also, based on the voice data group, the voice data is synthesized according to the position of the virtual camera, and integrated with the whole view image of the virtual viewpoint to generate a virtual viewpoint image and voice (S03308).

  The back-end server 270 transmits the generated virtual viewpoint image and sound to the virtual camera operation UI 330 (S03309). The virtual camera operation UI 330 displays a captured image of the virtual camera by displaying the received image.

  FIG. 38A is a flowchart illustrating a processing procedure when the virtual camera operation UI 330 generates a live image.

  In step S08201, the operator obtains operation information input to the input device to operate the virtual camera 08001. In step S08202, the virtual camera operation unit 08101 determines whether the operator's operation is movement or rotation of the virtual camera 08001. The movement and rotation here are performed for each frame. If it is determined that the operation is movement or rotation, the process advances to step S08203. Otherwise, the process advances to step S08205. Here, the processing branches depending on the movement operation, the rotation operation, and the trajectory selection operation. Thus, there is an effect that it is possible to switch between the image expression in which the time is stopped and the viewpoint position is rotated and the image expression in which the continuous movement is expressed by a simple operation.

  In S08203, the processing for one frame described in FIG. 38B is performed. In step S08204, the virtual camera operation UI 330 determines whether the operator has input an end operation. If an end operation has been input, the process ends. If the processing has not been completed, the process returns to S08201.

  Next, in step S08205, the virtual camera operation unit 08101 determines whether or not the operator has input a trajectory (virtual camera path) selection operation. For example, the trajectory can be represented by a row of operation information of the virtual camera 08001 for a plurality of frames. If it is determined that the trajectory selection operation has been input, the process advances to step S08206. If not, the process returns to S08201.

  In step S08206, the virtual camera operation UI 330 acquires the operation of the next frame from the selected trajectory. In S08207, the processing for one frame described in FIG. 38B is performed. In step S08208, it is determined whether the processing for all frames of the selected trajectory has been completed. If the processing has been completed, the process advances to step S08204. If not, the process returns to S08206.

  FIG. 38B is a flowchart illustrating the processing for one frame in S08203 and S08206.

  In step S08209, the virtual camera parameter deriving unit 08102 derives virtual camera parameters after the position and orientation have been changed. In S08210, the collision determination unit 08104 performs a collision determination. If the collision has occurred, that is, if the virtual camera constraint has not been satisfied, the process advances to step S08214. If there is no collision, that is, if the virtual camera constraint is satisfied, the process advances to step S08211.

  Thus, the collision determination is performed in the virtual camera operation UI 330. Then, according to the determination result, for example, a process of locking the operation unit or displaying a message of a different color to give a warning is performed, so that the immediateness of feedback to the operator can be improved. It is clearly stated that this has the effect of improving the operability of the operator as a result.

  In step S08211, the virtual camera path management unit 08106 transmits the virtual camera parameters to the back-end server 270. In step S08212, the virtual camera image / sound output unit 08108 outputs the image received from the back-end server 270.

  In step S08214, the position and orientation of the virtual camera 08001 are corrected so as to satisfy the virtual camera constraint. For example, the latest operation input by the user is canceled, and the virtual camera parameters are returned to the state one frame before. This allows the operator to interactively correct the operation input from the place where the collision occurred without re-executing the operation input from the beginning, for example, when a collision occurs due to a trajectory input being performed. There is an effect to be improved.

  In step S08215, the feedback output unit 08105 notifies the operator that the virtual camera restriction is not satisfied. The notification is performed by a sound, a message, a method of locking the virtual camera operation UI 330, or the like. The notification method is not limited to this.

  FIG. 39 is a flowchart illustrating a processing procedure when the virtual camera operation UI 330 generates a replay image.

  In step S08301, the virtual camera path management unit 08106 acquires the virtual camera path 08002 of the live image. In step S08302, the virtual camera path management unit 08106 receives an operator operation for selecting a start point and an end point from the virtual camera path 08002 of the live image. For example, a virtual camera path 08002 for 10 seconds before and after the goal scene can be selected. When the live image is 60 frames / sec, 600 virtual camera parameters are included in the virtual camera path 08002 for 10 seconds. In this way, the virtual camera parameter information is managed in association with each frame.

  In step S08303, the selected virtual camera path 08002 for 10 seconds is stored as the initial value of the virtual camera path 08002 of the replay image. When the virtual camera path 08002 is edited by the processing from S08307 to S08309, the edited result is overwritten and saved.

  In step S08304, the virtual camera operation UI 330 determines whether the operation input by the operator is a reproduction operation. In the case of a reproduction operation, the process proceeds to S08305. If it is not a reproduction operation, the process proceeds to S08307.

  In step S08305, an operator input relating to selection of a reproduction range is received. In step S08306, images and sounds in the range selected by the operator are reproduced. Specifically, the virtual camera path management unit 08106 transmits the virtual camera path 08002 in the selected range to the back-end server 270. That is, the virtual camera parameters included in the virtual camera path 08002 are sequentially transmitted. Then, the virtual camera image / sound output unit 08108 outputs the virtual viewpoint image and the virtual viewpoint sound received from the back-end server 270.

  In step S08307, the virtual camera operation UI 330 determines whether the operation input by the operator is an editing operation. In the case of editing, the process proceeds to S08308. If not, the process advances to step S08310.

  In step S08308, the virtual camera operation UI 330 specifies a range selected by the operator as the editing range. In S08309, the image and sound in the selected editing range are reproduced by the same processing as in S08306. However, at this time, when the virtual camera 08001 is operated using the virtual camera operation unit 08101, the result is reflected. That is, it is possible to edit the replay image so that the image has a different viewpoint from the live image. Further, the replay image may be edited so as to perform slow reproduction or stop. For example, editing in which time is stopped and the viewpoint is moved is also possible.

  In S08310, the virtual camera operation UI 330 determines whether the operation input by the operator is an end operation. In the case of termination, the process proceeds to S083111. If not, the process advances to step S08304.

  In step S08311, the edited virtual camera path 08002 is transmitted to the back-end server 270.

  FIG. 40 is a flowchart illustrating a processing procedure for a user to select and view a desired virtual camera image from a plurality of virtual camera images generated using the virtual camera operation UI 330. For example, the user uses the end user terminal 190 to view a virtual camera image. The virtual camera path 08002 may be stored in the image computing server 200 or may be stored in a different Web server (not shown).

  In step S08401, the end user terminal 190 acquires a list of virtual camera paths 08002. Each virtual camera path 08002 may have a thumbnail, a user evaluation, and the like added thereto. In step S08401, a list of virtual camera paths 08002 is displayed on the end user terminal 190.

  In step S08402, the end user terminal 190 acquires designation information on the virtual camera path 08002 selected by the user from the list.

  In step S08403, the end user terminal 190 transmits the virtual camera path 08002 selected by the user to the back-end server 270. The back-end server 270 generates a virtual viewpoint image and a virtual viewpoint sound from the received virtual camera path 08002 and transmits them to the end user terminal 190.

  In S08404, the end user terminal 190 outputs the virtual viewpoint image and the virtual viewpoint sound received from the back-end server 270.

  As described above, by storing the list of virtual camera paths and enabling the image to be reproduced later using the virtual camera path, it is not necessary to continuously store the virtual viewpoint images, and the cost of the storage device can be reduced. Will be possible. Furthermore, when an image generation of a virtual camera path with a high priority is requested, the order of generating a virtual camera path image with a low priority can be dealt with later. Further, when the virtual camera path is made public on a Web server, it is possible to provide or share a virtual viewpoint image to an end user connected to the Web, thereby improving serviceability to the user. Here, it is specified that the effect is obtained.

  The screen displayed on the end user terminal 190 will be described. FIG. 41 is an example of a display screen 41001 displayed by the end user terminal 190.

  The end user terminal 190 sequentially displays the image input from the back-end server 270 in the area 41002 where the image is displayed, so that the viewer (user) can view a virtual viewpoint image such as a soccer game. Become. The viewer switches the viewpoint of the image by operating the user input device according to the displayed image. For example, when the user moves the mouse to the left, an image whose viewpoint is directed to the left in the displayed image is displayed. When the mouse is moved upward, an image in which the upward direction of the displayed image is looked up is displayed.

  In an area other than the image display area 41002, a GUI (Graphic User Interface) button 41003 and a button 41004 which can switch between manual control and automatic control are provided. By performing an operation corresponding to this, the viewer can select whether to change the viewpoint by himself / herself for viewing or to view from a preset viewpoint.

  For example, a certain end-user terminal 190 sequentially uploads viewpoint operation information indicating a result of viewpoint switching by manual operation of a user to the image computing server 200 or a Web server (not shown). Then, a user who operates another end user terminal 190 can obtain the viewpoint operation information and view a virtual viewpoint image corresponding to the information. In addition, by enabling rating of the viewpoint operation information to be uploaded, the user can select and view an image corresponding to, for example, popular viewpoint operation information, so that even a user unfamiliar with the operation can easily use the service. There is a special effect that it can be used.

  Next, the operation of the application management unit 10001 when the viewer selects manual control and performs manual control will be described. FIG. 42 is a flowchart illustrating the manual control processing of the application management unit 10001.

  The application management unit 10001 determines whether there is an input by the user (S10010).

  If there is an input by the user (Yes in S10010), the application management unit 10001 converts the user input information into a back-end server command that can be recognized by the back-end server 270 (S10011).

  On the other hand, when there is no input by the user (No in S10010), the process proceeds to S10013.

  Next, the application management unit 10001 transmits the back-end server command via the basic software unit 10002 and the network communication unit 10003 (S10012).

  After the back-end server 270 generates the image whose viewpoint has been changed based on the user input information, the application management unit 10001 receives the image from the back-end server 270 via the network communication unit 10003 and the basic software unit 10002 (S10013). . Then, the application management unit 10001 displays the received image in a predetermined image display area 41002 (S10014). By performing the above processing, the viewpoint of the image is changed by manual operation.

  Next, the operation of the application management unit 10001 when the viewer (user) selects the automatic control will be described. FIG. 43 is a flowchart showing the automatic control process of the application management unit 10001.

  When there is input information for automatic driving (S10020), the application management unit 10001 reads out input information for automatic driving (S10021).

  The application management unit 10001 converts the read automatic control input information into a back-end server command that can be recognized by the back-end server 270 (S10022).

  Next, a back-end server command is transmitted via the basic software unit 10002 and the network communication unit 10003 (S10023).

  After the back-end server 270 generates the image whose viewpoint has been changed based on the user input information, the application management unit 10001 receives the image from the back-end server 270 via the network communication unit 10003 and the basic software unit 10002 (S10024). . Finally, the application management unit 10001 displays the received image in a predetermined image display area (S10025). By repeating the above process as long as there is input information for automatic driving, the viewpoint of the image is changed by the automatic driving.

  FIG. 44 shows a processing flow in which the back-end server 270 generates a virtual viewpoint image of one frame.

  First, the data receiving unit 03001 receives virtual camera parameters from the controller 300 (S03100). As described above, the virtual camera parameters are data representing the position and orientation of the virtual viewpoint.

  The foreground object determination unit 03010 determines a foreground object required for generating a virtual viewpoint image based on the received virtual camera parameters and the position of the foreground object (S03101). A foreground object that enters the field of view when viewed from a virtual viewpoint is obtained in a three-dimensional geometric manner. The request list generation unit 03011 generates a request list of the determined foreground image, foreground three-dimensional model group, background image, and audio data group of the foreground object, and makes a request to the database 250 from the request data output unit 03012 (S03102). The request list is the content of data requested to the database 250.

  The data receiving unit 03001 receives the requested information from the database 250 (S03103). The data receiving unit 03001 determines whether the information received from the database 250 includes information indicating an error (S03104).

Here, examples of the information indicating the error include an overflow of an image transfer amount, a failure in image capturing, a failure in saving an image in a database, and the like. This error information is stored in the database 250.
If information indicating an error is included in S03104, the data receiving unit 03001 determines that generation of a virtual viewpoint image is not possible, and ends the process without outputting data.

  If the information indicating the error is not included in S03104, the back-end server 270 performs the generation of the background image in the virtual viewpoint, the generation of the foreground image, and the generation of the sound according to the viewpoint. The background texture pasting unit 03002 generates a textured background mesh model from the background mesh model acquired after the system startup and held by the background mesh model management unit 03013 and the background image acquired from the database 250 (S03105).

  The back-end server 270 generates a foreground image according to the rendering mode (S03106). Further, the back-end server 270 synthesizes a sound data group so as to simulate how the sound is heard from the virtual viewpoint, and generates sound (S03107). In synthesizing the audio data group, the size of each audio data to be synthesized is adjusted based on the virtual viewpoint and the acquisition position of the audio data.

  The rendering unit 03006 trims the textured background mesh model generated in S3105 to the visual field viewed from the virtual viewpoint, and combines this with the foreground image to generate a full-view image of the virtual viewpoint (S03108).

The synthesizing unit 03008 integrates the virtual sound generated by the virtual viewpoint sound generation (S03107) and the rendered full-view image of the virtual viewpoint (S03109), and generates one frame of virtual viewpoint contents.
The image output unit 03909 outputs the generated one-frame virtual viewpoint content to the external controller 300 and the end user terminal 190 (S03110).

  Next, in order to increase use cases to which the present system can be applied, a description will be given of performing flexible control determination that can respond to various requests for generating virtual viewpoint images.

  FIG. 45 shows a flow of generating a foreground image. Here, an example of a selection guideline for selecting any one of a plurality of rendering algorithms in order to respond to a request corresponding to an image output destination in generating a virtual viewpoint image will be described.

  First, the rendering mode management unit 03014 of the back-end server 270 determines a rendering method. The requirements for determining the rendering method are set from the control station 310 to the back-end server 270. The rendering mode management unit 03014 determines a rendering method according to requirements. The rendering mode management unit 03014 checks whether a request to prioritize high speed has been made in the generation of the virtual viewpoint image in the back-end server 270 based on the image captured by the camera 112 (S03200). A request that gives priority to high speed is equivalent to a request for image generation with low delay. In the case of YES in S03200, IBR is enabled as rendering (S03201). Next, it is confirmed whether or not a request for giving priority to the degree of freedom in designating a viewpoint related to virtual viewpoint image generation has been made (S03202). In the case of YES in S03202, the MBR is enabled as rendering (S03203). Next, it is confirmed whether or not a request to prioritize reduction in calculation processing has been made in the generation of the virtual viewpoint image (S03204). A request to give priority to the reduction in computational processing is made, for example, when a system is configured at low cost without using much computer resources. In the case of YES in S03204, IBR is enabled as rendering (S03205). Next, the rendering mode management unit 03014 checks whether or not the number of cameras 112 used for generating the virtual viewpoint image is equal to or larger than a threshold (S03206). In the case of YES in S03206, MBR is enabled as rendering (S03207).

  The back-end server 270 determines whether the rendering method is the MBR or the IBR based on the mode information managed by the rendering mode management unit 03014 (S03208). When none of the processes of S03201, S03203, S03205, and S03207 is performed, a default rendering method determined in advance when the system is operating is used.

  If it is determined in S03208 that the rendering method is model-based (MBR), the foreground texture determination unit 03003 determines the foreground texture based on the foreground three-dimensional model and the foreground image group (S03209). Then, the foreground texture boundary color matching unit 03004 performs color matching of the determined foreground texture boundary (S03210). Since the texture of the foreground three-dimensional model is extracted from a plurality of foreground image groups, this color matching is performed in response to the difference in texture color due to the difference in the shooting state of each foreground image.

  If it is determined in step S03208 that the rendering method is IBR, the virtual viewpoint foreground image generation unit 03005 performs geometric transformation such as perspective transformation on each foreground image based on the virtual camera parameters and the foreground image group, and outputs the foreground image from the virtual viewpoint. Is generated (S03211).

  The user may arbitrarily change the rendering method while the system is operating, or the system may change the rendering method according to the state of the virtual viewpoint. Further, the rendering method that is a candidate may be changed during operation of the system.

  Thus, the rendering algorithm related to the generation of the virtual viewpoint image can be changed according to the situation, in addition to being set at the time of activation, so that various requests can be met. That is, even if the image output destination requests different requirements (for example, the priority of each parameter), there is an effect that it is possible to flexibly respond. In the present embodiment, it is assumed that one of the IBR and the MBR is used as the rendering method. However, the present invention is not limited to this, and a hybrid method using both methods may be used. In the case of using the hybrid method, the rendering mode management unit 03014 determines a plurality of generation methods used to generate each of the plurality of divided regions obtained by dividing the virtual viewpoint image, based on the information acquired by the data reception unit 03001. That is, a part of the one-frame virtual viewpoint image may be generated based on the MBR, and another part of the virtual viewpoint image may be generated based on the IBR. For example, for objects such as glossy, textureless, and non-convex surfaces, the IBR is used to avoid a decrease in the accuracy of the three-dimensional model, and for objects close to the virtual viewpoint, the image becomes planar by using the MBR. There is a method such as avoiding the situation. Further, for example, an object near the center of the screen is desired to be displayed clearly, so that an image is generated by MBR, and an object at an end is generated by IBR, so that the processing load can be reduced. This makes it possible to control the processing load for generating the virtual viewpoint image and the image quality of the virtual viewpoint image in more detail.

  In addition, appropriate settings of the system such as gazing point, camera work, and transmission control may differ depending on the competition, but if the operator manually sets the system every time the competition is held, the operator Since there is a risk that the time and labor of the operation may increase, simplification of the setting is required. Therefore, the image processing system 100 provides a mechanism for automatically updating the setting of the device for which the setting is to be changed, thereby reducing the trouble of the operator who sets the system for generating the virtual viewpoint image. This mechanism will be described below.

  FIG. 46 is an information list related to operations generated in the above-described post-installation workflow and set in the devices configuring the system in the pre-imaging workflow. The control station 310 acquires competition information on a competition to be photographed by the plurality of cameras 112 based on an input operation by a user. Note that the method of acquiring the competition information is not limited to this, and for example, the control station 310 may acquire the competition information from another device. Then, the control station 310 associates the acquired competition information with the setting information of the image processing system 100 and holds the information as the information list. Hereinafter, the information list regarding the operation is referred to as a setting list. The control station 310 operates as a control device that performs a system setting process based on the held setting list, thereby reducing the time and effort of an operator who sets the system.

  The competition information acquired by the control station 310 includes, for example, at least one of the type of competition to be photographed and the start time. However, the competition information is not limited to this, and may be other information regarding the competition.

  The shooting number 46101 represents a scene corresponding to each competition to be photographed, and the scheduled time 46103 is a scheduled start time and a scheduled end time of each competition. Before the start time of each scene, the control station 310 issues a change request to each device according to the setting list.

  The competition name 46102 is the name of the competition type. The gazing point (coordinate designation) 46104 includes the number of gazing points of the cameras 112a to 112z, the coordinate position of each gazing point, and the camera number corresponding to each gazing point. The shooting direction of each camera 112 is determined according to the position of the gazing point.

  The camera work 46105 indicates a range of a camera path when a virtual viewpoint is operated by the virtual camera operation UI 330 and the back-end server 270 to generate an image. Based on the camera work 46105, a specifiable range of a viewpoint related to generation of a virtual viewpoint image is determined.

  The calibration file 46106 is a file that stores camera parameter values relating to the alignment of the plurality of cameras 112 related to generation of the virtual viewpoint image, which are derived in the installation calibration described with reference to FIG. Generated.

  The image generation algorithm 46107 indicates a setting as to which of IBR, MBR, and a hybrid method using both is used as a rendering method for generating a virtual viewpoint image based on a captured image. The rendering method is set from the control station 310 to the back-end server 270. For example, a virtual viewpoint image is generated using competition information indicating a type of a competition corresponding to a number of players equal to or less than a threshold, such as a shot put or a high jump with a shooting number of 3, and a three-dimensional model generated based on the shot image. And the setting information indicating the MBR scheme to be performed. As a result, the degree of freedom in specifying the viewpoint in the virtual viewpoint image of the competition with few participating players increases. On the other hand, in a competition in which the number of participating players is large, such as the opening ceremony of the shooting number = 1, the processing load increases when trying to generate a virtual viewpoint image by the MBR method, so that the virtual viewpoint image is generated with a smaller processing load. Possible IBR methods are associated.

  The foreground / background transmission 46108 indicates the settings of the compression ratio and the frame rate (unit: fps) for each of the foreground image (represented by FG) and the background image (represented by BG) separated from the captured image. Note that the foreground image is a foreground image generated based on the foreground region extracted from the captured image for generation of the virtual viewpoint image and transmitted in the image processing system 100, and the background image is similarly extracted from the captured image. 3 is a background image generated and transmitted based on a background area.

  FIG. 47 shows an operation sequence in a case where information relating to the photographing number = 2 in the setting list is set in the apparatus constituting the system in the pre-imaging workflow by the control station 310.

  After the start of the system operation, the control station 310 confirms the scheduled start time of the game to be photographed which is specified from the held setting list (F47101). Then, when it is a predetermined time before the scheduled start time, the control station 310 starts the setting process corresponding to the shooting number = 2 (F47102). The predetermined time is, for example, the time required for the setting process. In this way, by automatically starting the setting process a predetermined time before the start of the game, the setting can be completed at the start of the game without the operator instructing the start of the setting. When the user instructs to start the setting, the control station 310 may start the setting process regardless of the start time of the game.

  The setting process performed by the control station 310 includes, for example, a process of setting parameters related to image processing of a device that generates a virtual viewpoint image, a process of setting parameters related to shooting of the plurality of cameras 112, and the like. However, the content of the setting process is not limited to these, and may be, for example, a process of activating an apparatus included in the image processing system 100.

  First, the control station 310 sets a gazing point (F47103). Then, a request is made to the camera adapter 120 to set the gazing point coordinates for each camera (F47104). Here, the cameras 112 are grouped according to the gazing points, and gazing points having the same coordinates are set to the cameras 112 in each gazing point group. When the camera adapter 120 receives the request for setting the gazing point coordinates for each camera, the camera adapter 120 includes a pan / tilt (referred to as PT) setting instruction for the camera platform 113 and a setting instruction such as lens tilt for the camera 112 and the lens. A table PT instruction request is transmitted (F47105). The processing of F47104 and F47105 is repeatedly executed for the number of sensor systems 110. Further, the control station 310 sets gazing point group information for each camera in the front-end server 230 and the database 250 (F47106).

  Next, the control station 310 sets the values acquired by the calibration (F47107). That is, the information of the calibration file is set for all the sensor systems 110. Then, the control station 310 transmits a calibration setting request to each camera adapter 120 (F47108). Upon receiving this, the camera adapter 120 sets shooting parameters, focus, and zoom for the camera 112 lens and the camera platform 113 (F47109). Further, the control station 310 also issues a calibration setting request to the front-end server 230 (F47110).

  Next, the control station 310 sets camera work (F47111). Then, the control station 310 requests the back-end server 270 to set the camera groups grouped according to the point of interest, the shooting range of each camera 112, and the range of the virtual camera path (F47112). The back-end server 270 needs information related to camera work in order to map the viewpoint path of the virtual camera 08001 from the virtual camera operation UI 330 to the physical camera 112 and render an image. The back-end server 270 issues a virtual camera trial request to the virtual camera operation UI 330 to check the movable range of the virtual camera (F47113). Then, the back-end server 270 receives the virtual camera operation notification from the virtual camera operation UI 330 (F47114). Here, the back-end server 270 determines that there is no valid image corresponding to the viewpoint position corresponding to the received virtual camera operation notification (F47115). Then, the back-end server 270 notifies the virtual camera operation UI 330 of an error (F47116). The virtual camera operation UI 330 determines that further viewpoint movement cannot be performed in response to the error notification, performs a virtual camera operation to another viewpoint again, and notifies the back-end server 270 (F47117). The back-end server 270 confirms that there is an effective image corresponding to the viewpoint corresponding to the notification (F47118), and transmits a corresponding image response to the virtual camera operation UI 330 (F47119).

  Next, the control station 310 sets an image generation algorithm (F47120). Then, it instructs the back-end server whether to use the IBR, MBR, or hybrid algorithm (F47121).

  Next, the control station 310 sets the transmission method of the foreground image and the background image (F47112). Then, the control station 310 sets the compression ratio of the foreground image (FG) and the background image (BG) and the frame rate (the number of frames per second: fps) to the camera adapter 120 based on the setting list. . In this figure, it is instructed to the camera adapter 120 that the compression ratio of the FG is 1/3 compression, the frame rate of the FG is 60 fps, and BG is not transmitted (F47123). In this case, since the background image is not sent from the camera adapter 120, the back-end server 270 cannot obtain the background texture at the time of rendering. Therefore, the control station 310 instructs the back-end server 270 to use the background three-dimensional model, that is, an instruction to generate a background image based on a stadium-shaped wire frame (F47124).

  With the above processing performed, shooting is performed until the end time of the competition. Note that the competition time may be extended, so that the operator may make a final decision to stop shooting.

  After the end of the shooting, the control station 310 performs the system start processing before the scheduled start time of the next scene. That is, the control station 310 confirms the scheduled start time of the scene with the photographing number = 3 (F47125), and performs the setting related to the photographing number = 3 for each device constituting the system (F47126). Thereafter, the processing described above is repeatedly performed according to the setting list.

  As described above, since the control station 310 automatically sets each device, the operator only has to perform a system start operation, a confirmation operation, and the like, and the operator's work related to complicated shooting control can be simplified. effective.

  FIG. 48 is a flowchart showing the reception control of the camera synchronous image frame m received by the front end server 230 from the camera adapter 120 via the daisy chain lane. In a case where the daisy chain is configured every half lap of the stadium or extends over a plurality of floors, if the front-end server 230 waits until the reception of the image data of all the cameras 112 is completed, a low-delay virtual viewpoint image generation is performed. May be difficult to achieve. Hereinafter, control for reducing this risk will be described.

  First, the front end server 230 receives an image data packet for each camera lane in the daisy chain (S48100). Then, the camera synchronous image frame m is sequentially stored (S48101). Next, it is confirmed whether there is one gazing point group (S48102). If NO in S48102, that is, if there are a plurality of gazing point groups, the camera image frames are classified into the plurality of gazing point groups (S48103). Next, the front-end server 230 checks whether or not there is a gazing point group for which reception of the image frame m has been completed among the camera groups 112 for each gazing point (S48104). If there is a point of interest group that has been received, image processing is performed in the image processing unit 02150, the three-dimensional model coupling unit 02160, the image coupling unit 2170, and the photographing data file generation unit 02180 for each point of interest group (S48105). . Next, the front-end server 230 determines whether or not the image processing has been completed for all the gazing point groups. If the image processing has not been completed (NO in S48106), the front-end server 230 checks whether a time-out for waiting for an image frame has occurred (S48107). The timeout threshold may be determined according to a unit time per frame. In the case of YES in S48107, the front-end server 230 detects the lost image frame, marks information indicating the loss on the target frame of the camera 112 in which the loss has occurred (S48108), and writes the image data into the database 250. Accordingly, the back-end server 270 can recognize the loss of the image frame, and is effective in performing the rendering process. That is, when the back-end server 270 performs mapping between the virtual camera specified by the virtual camera operation UI 330 and the real camera 112, the back-end server 270 can immediately determine the image of the lost camera 112. Therefore, in the case where the generated virtual viewpoint image may be broken, there is an effect that the image output can be automatically corrected without relying on the visual observation of the operator.

  Subsequently, the hardware configuration of each device configuring the present embodiment will be described in more detail. As described above, the present embodiment has mainly described an example in which the camera adapter 120 is implemented with hardware such as an FPGA and / or an ASIC, and the hardware executes the above-described processing. The same applies to various devices in the sensor system 110, the front-end server 230, the database 250, the back-end server 270, and the controller 300. However, at least one of the above devices may execute the processing of the present embodiment by software processing using, for example, a CPU, GPU, DSP, or the like.

  FIG. 49 is a block diagram showing a hardware configuration of the camera adapter 120 for realizing the functional configuration shown in FIG. 2 by software processing. Note that devices such as the front-end server 230, the database 250, the back-end server 270, the control station 310, the virtual camera operation UI 330, and the end user terminal 190 can also have the hardware configuration of FIG. The camera adapter 120 includes a CPU 1201, a ROM 1202, a RAM 1203, an auxiliary storage device 1204, a display unit 1205, an operation unit 1206, a communication unit 1207, and a bus 1208.

  The CPU 1201 controls the entire camera adapter 120 using computer programs and data stored in the ROM 1202 and the RAM 1203. The ROM 1202 stores programs and parameters that do not need to be changed. The RAM 1203 temporarily stores programs and data supplied from the auxiliary storage device 1204, data supplied from the outside via the communication unit 1207, and the like. The auxiliary storage device 1204 includes, for example, a hard disk drive and stores content data such as still images and moving images.

  The display unit 1205 includes, for example, a liquid crystal display, and displays a GUI (Graphical User Interface) for a user to operate the camera adapter 120 and the like. The operation unit 1206 includes, for example, a keyboard and a mouse, and inputs various instructions to the CPU 1201 in response to an operation by a user. The communication unit 1207 communicates with external devices such as the camera 112 and the front-end server 230. For example, when the camera adapter 120 is connected to an external device by wire, a LAN cable or the like is connected to the communication unit 1207. When the camera adapter 120 has a function of performing wireless communication with an external device, the communication unit 1207 includes an antenna. A bus 1208 connects the components of the camera adapter 120 to transmit information.

  Note that, for example, part of the processing of the camera adapter 120 may be performed by the FPGA, and another part of the processing may be realized by software processing using a CPU. Further, each component of the camera adapter 120 shown in FIG. 49 may be configured by a single electronic circuit, or may be configured by a plurality of electronic circuits. For example, the camera adapter 120 may include a plurality of electronic circuits that operate as the CPU 1201. When the plurality of electronic circuits perform processing as the CPU 1201 in parallel, the processing speed of the camera adapter can be improved.

  Further, in the present embodiment, the display unit 1205 and the operation unit 1206 exist inside the camera adapter 120, but the camera adapter 120 may not include at least one of the display unit 1205 and the operation unit 1206. Further, at least one of the display unit 1205 and the operation unit 1206 exists as another device outside the camera adapter 120, and the CPU 1201 controls the display control unit that controls the display unit 1205 and the operation control unit that controls the operation unit 1206. It may operate as a unit.

  The same applies to other devices in the image processing system 100. For example, the front-end server 230, the database 250, and the back-end server 270 may not include the display unit 1205, and the control station 310, the virtual camera operation UI 330, and the end user terminal 190 may include the display unit 1205. Further, the above-described embodiment has been described centering on an example in which the image processing system 100 is installed in a facility such as a stadium or a concert hall. Other examples of facilities include, for example, amusement parks, parks, racetracks, bicycle racetracks, casinos, pools, skating rinks, ski resorts, live houses, and the like. In addition, events performed in various facilities may be performed indoors or outdoors. The facilities in the present embodiment also include facilities that are temporarily constructed (for a limited time).

  The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  As described above, according to the above-described embodiment, it is possible to easily generate a virtual viewpoint image regardless of the scale of a device configuring the system such as the number of cameras 112, the output resolution of a captured image, the output frame rate, and the like. . As described above, the embodiments of the present invention have been described in detail. However, the present invention is not limited to the above embodiments, and various modifications and changes may be made within the scope of the present invention described in the appended claims. Is possible.

110a Sensor system 111a Microphone 112a Camera 113a Pan head 120a Camera adapter 180 Switching hub 190 End user terminal 230 Front end server 250 Database 270 Back end server 290 Time server 310 Control station 330 Virtual camera operation UI

Claims (21)

A control device that performs setting processing of a system that generates a virtual viewpoint image based on captured images acquired by capturing from a plurality of directions,
Holding control means for holding the plurality of system setting information configured by the plurality of operation information in the holding means,
Setting means for performing a system setting process based on one system setting information of the plurality of system setting information held by the holding means,
The plurality of operation information includes information about a specific area to which a plurality of imaging devices are directed,
The control device, wherein the setting process includes a setting process for directing a plurality of photographing devices to the specific area. The apparatus further includes a receiving unit that receives an instruction of a system setting process from a user,
The setting unit performs a system setting process based on an instruction from a user received by the receiving unit and one of a plurality of system setting information held by the holding unit. The control device according to claim 1.   The control device according to claim 1, wherein the holding control unit causes the holding unit to hold a setting list in which a plurality of system setting information is listed.   The control device according to any one of claims 1 to 3, wherein the setting process further includes a setting process of setting parameters related to photographing of a plurality of photographing devices. Before Ki設 constant process, the control device according to any one of claims 1 to 4, further comprising a setting process of setting the parameters relating to image processing apparatus for generating the virtual viewpoint image.   The control device according to any one of claims 1 to 5, wherein the setting process further includes a setting process for activating a device included in the system.   The control device according to claim 1, wherein the plurality of pieces of operation information further include information indicating a start time of the event to be captured.   The control device according to any one of claims 1 to 7, wherein the plurality of operation information further includes information indicating a parameter regarding alignment of the plurality of imaging devices related to generation of the virtual viewpoint image. .   The control device according to claim 1, wherein the plurality of pieces of operation information further include information on a specifiable range of a viewpoint related to generation of the virtual viewpoint image.   The control device according to claim 1, wherein the plurality of operation information further includes information on a method of generating the virtual viewpoint image based on the captured image. The method according to claim 1, wherein the plurality of pieces of operation information further include information on a method of transmitting a foreground image in the system based on a foreground area extracted from the captured image. Control device. 12. The control device according to claim 11, wherein the information on the transmission method of the foreground image in the system includes at least one of information indicating a compression rate of the foreground image and information indicating a frame rate. 13. The system according to claim 1, wherein the plurality of operation information further includes information on a method of transmitting a background image in the system based on a background region extracted from the captured image. Control device. 14. The control device according to claim 13, wherein the information on the transmission method of the background image in the system includes at least one of information indicating a compression rate of the background image and information indicating a frame rate.   The information on the specific area includes at least one of information indicating the number of the specific area, information indicating a position of the specific area, and identification information of an imaging device directed to the specific area. The control device according to claim 1, wherein: A control method performed by a control device in a system that generates a virtual viewpoint image based on captured images obtained by capturing from a plurality of directions,
A holding control step of holding a plurality of system setting information configured by a plurality of operation information in a holding unit,
A setting step of performing a system setting process based on one system setting information of the plurality of system setting information held by the holding unit,
The plurality of operation information includes information about a specific area to which a plurality of imaging devices are directed,
The control method according to claim 1, wherein the setting process includes a setting process of pointing a plurality of photographing devices to the specific area. Further comprising a receiving step of receiving a system setting instruction from the user,
In the setting step, a system setting process is performed based on an instruction from a user received in the receiving step and one of a plurality of system setting information held by the holding unit. The control method according to claim 16, characterized in that:   18. The control method according to claim 16, wherein the plurality of pieces of operation information further include information indicating a start time of the event to be photographed.   The control method according to any one of claims 16 to 18, wherein the plurality of pieces of operation information further include information on a specifiable range of a viewpoint related to generation of the virtual viewpoint image.   20. The control method according to claim 16, wherein the plurality of pieces of operation information further include information on a method for generating the virtual viewpoint image based on the captured image.   A program for causing a computer to operate as each unit of the control device according to claim 1.

Images