JP7042571B2 - Image processing device and its control method, program - Google Patents

Description

The present invention relates to an image processing system including a plurality of cameras for photographing a subject from a plurality of directions.

Recently, a technique of installing a plurality of cameras at different positions to perform synchronous shooting from multiple viewpoints and generating virtual viewpoint contents using the multi-viewpoint images obtained by the shooting has attracted attention. According to such a technique, for example, a highlight scene of soccer or basketball can be viewed from various angles, so that a user can be given a high sense of presence as compared with a normal image.

In recent years, the amount of data output by a camera has been increasing with the improvement of camera performance such as resolution and frame rate. When the image taken by the camera has a high resolution and a high frame rate, the data transmission capacity becomes large, so that the transmission speed is affected by the band.

Conventional techniques include a technique for compressing and coding a captured image and a technique for controlling a frame rate based on a data capacity. For example, in Patent Document 1, in order to transmit a plurality of image signals transmitted from a plurality of camera systems in a limited transmission band, an important area determination unit weights the frame rate of the image signals for each camera. , A configuration for controlling the frame rate as a whole is disclosed.

Japanese Unexamined Patent Publication No. 2007-221367

There is a demand to generate a slow-playing replay image for determining sports competitions using virtual viewpoint contents. To meet the above requirements, high-capacity data with a high frame rate is required, but as the number of cameras increases, it is difficult to transmit all of the plurality of cameras at a high frame rate in a practical time.

The present invention has been made in view of the above problems, and an object of the present invention is to enable an appropriate setting of a frame rate of a captured image captured by a plurality of imaging devices that capture a subject from different directions. ..

The image processing apparatus according to one aspect of the present invention has the following configuration. That is,
Specific means to identify the position of the object of interest taken by multiple shooting devices,
When it is specified by the specific means that the position of the object of interest is included in the specific area, the image transmitted by one or more image pickup devices having a predetermined positional relationship with the specific area among the plurality of image pickup devices. The plurality of imaging devices are controlled so that the frame rate of the data is higher than the frame rate of the image data transmitted by another imaging device different from the one or more imaging devices among the plurality of imaging devices. It has a control means.

According to the present invention, it becomes possible to appropriately set the frame rate of a captured image captured by a plurality of imaging devices that capture a subject from different directions.

The figure for demonstrating the structure of the image processing system 100. The block diagram for demonstrating the functional configuration of a camera adapter 120. The block diagram for demonstrating the functional configuration of the back-end server 270. The figure for demonstrating the operation of the camera selection part 3015. Diagram to illustrate the camera selected. The figure for demonstrating the virtual camera 8001. The block diagram for demonstrating the functional configuration of the virtual camera operation UI 330. The flowchart which shows the processing of the camera selection part 3015. The flowchart which shows the processing of the virtual viewpoint foreground image generation part 3005. The block diagram which shows the hardware composition of a camera adapter 120. The block diagram for demonstrating the functional configuration of the back-end server 270. The figure for demonstrating the operation of the camera selection part 3016. Diagram to illustrate the camera selected.

Prior to detailing the embodiment of the present invention, there is a problem in generating a slow-playing replay image for determining a sports competition in a system that generates a virtual viewpoint image using a plurality of cameras. Further explanation will be given.

In the generation of virtual viewpoint content based on a plurality of viewpoint images, a virtual viewpoint image is generated. That is, images taken by a plurality of cameras are aggregated in an image processing unit such as a server, and the image processing unit performs processing such as three-dimensional model generation and rendering to generate a virtual viewpoint image. Then, by transmitting this virtual viewpoint image to the user terminal, the user can view the virtual viewpoint content.

Here, as described above, in order to generate a replay image, high-capacity data with a high frame rate is required, but as the number of cameras increases, all of the multiple cameras can be used at a high frame rate in a practical time. It is difficult to transmit.

The technique disclosed in Patent Document 1 is suitable for a configuration in which images of a plurality of cameras are joined together to generate a wide range of images, but the same subject is photographed by a plurality of cameras and the images are used. It cannot be applied to this image processing system that generates virtual viewpoint images. That is, in the technique of Patent Document 1 in which the frame rate is changed for each camera, the control of each image is independent for connecting the images to generate a wide range of images. Therefore, it is possible to generate an image even if there are defects or omissions in the pixels when joining the images, and there is no problem even if the frame rates are different. On the other hand, in this system as described above, a virtual viewpoint image is generated from an image in which a plurality of cameras take the same subject at the same time. Therefore, if the images of all the cameras used for generating the virtual viewpoint image are not taken at the same time with high accuracy, there is a problem that the generated virtual viewpoint image is deteriorated.

Further, it is preferable that this system generates a virtual viewpoint image by mixing a live image for live broadcasting such as a sports competition and a replay image including slow playback. Therefore, it is necessary to generate a replay image at a faster frame rate without reducing the frame rate of the live image.

In order to solve the above problems, in the embodiment of the present invention, in a system for generating a virtual viewpoint image using a plurality of cameras, a plurality of captured images necessary for generating a high-quality virtual viewpoint image are high-framed. It provides a technology that can transmit at a frame rate.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown.
<Embodiment 1>
A system in which a plurality of cameras and microphones are installed in facilities such as a stadium and a concert hall to shoot and collect sound will be described with reference to FIG.

<Explanation of image processing system 100>
FIG. 1 is a diagram for explaining the configuration of the image processing system 100. The image processing system 100 includes a sensor system 110a, ..., 110z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

The controller 300 includes a control station 310 and a virtual camera operation UI 330. The control station 310 manages the operating state and controls parameter setting through the networks 310a, 310b, 310c, 180a, 180b, and 170a, ..., 170y for each block constituting the image processing system 100.

<Explanation of sensor system 110>
First, an operation of transmitting 26 sets of images and sounds of the sensor systems 110a, ..., The sensor system 110z from the sensor system 110z to the image computing server 200 will be described.

In the image processing system 100, the sensor systems 110a, ..., And the sensor system 110z are connected by a daisy chain. Here, in the present embodiment, unless otherwise specified, the 26 sets of systems from the sensor system 110a to the sensor system 110z are referred to as the sensor system 110 without distinction. Similarly, the devices in each sensor system 110 are not distinguished unless otherwise specified, and are described as a microphone 111, a camera 112, a pan head 113, an external sensor 114, and a camera adapter 120. The number of sensor systems is described as 26 sets, but this is just an example, and the number is not limited to this. Further, in the present embodiment, unless otherwise specified, the word "image" is described as including the concepts of moving images and still images. That is, the image processing system 100 of the present embodiment can process both still images and moving images. Further, in the present embodiment, an example in which the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual viewpoint sound will be mainly described, but the present invention is not limited to this. For example, the virtual viewpoint content may not include audio. Further, for example, the sound included in the virtual viewpoint content may be the sound collected by the microphone closest to the virtual viewpoint. Further, in the present embodiment, for the sake of simplification of the explanation, the description about the sound is partially omitted, but basically both the image and the sound are processed.

The sensor systems 110a, ..., 110z include one camera 112a, ..., 112z, respectively. That is, the image processing system 100 has a plurality of cameras for photographing the same subject from a plurality of directions. The optical axis of the camera 112 is fixed, and the camera 112 does not move left or right or up and down such as panning or tilting. The plurality of sensor systems 110 are connected to each other by a daisy chain. This connection form has the effect of reducing the number of connection cables and labor saving in wiring work in increasing the resolution of captured images to 4K or 8K and increasing the capacity of image data due to the increase in frame rate.

The sensor system 110 includes, but is not limited to, a microphone 111, a camera 112, a pan head 113, an external sensor 114, and a camera adapter 120.

The sound collected by the microphone 111a and the image captured by the camera 112a are subjected to the image processing described later in the camera adapter 120a and then transmitted to the camera adapter 120b of the sensor system 110b through the daisy chain 170a. .. Similarly, the sensor system 110b transmits the collected sound and the captured image to the sensor system 110c together with the image and the sound acquired from the sensor system 110a.

By continuing the above-mentioned operation, the images and sounds acquired by the sensor systems 110a, ..., 110z are transmitted from the sensor system 110z to the switching hub 180 using the network 180b, and then transmitted to the image computing server 200.

In the present embodiment, the cameras 112a, ..., 112z and the camera adapters 120a, ..., 120z are separated from each other, but they may be integrated in the same housing. In that case, the microphones 111a, ..., 111z may be built in the integrated camera 112 or may be connected to the outside of the camera 112.

<Explanation of Image Computing Server 200>
Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 processes the data acquired from the sensor system 110z.

The image computing server 200 includes a front-end server 230, a database 250 (hereinafter, may be referred to as a DB), a back-end server 270, and a time server 290.

The time server 290 has a function of distributing the time and synchronization signals, and distributes the time and synchronization signals to the sensor systems 110a, ..., 110z via the switching hub 180. The camera adapters 120a, ..., 120z that have received the time and synchronization signals cause the cameras 112a, ..., 112z to be Genlocked based on the time and the synchronization signal to perform image frame synchronization. That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112.

The front-end server 230 reconstructs a segmented transmission packet from the image and voice acquired from the sensor system 110z to convert the data format. After that, it is written in the database 250 according to the identifier of the camera, the data type indicating whether it is an image or an audio, and the frame number.

The front-end server 230 temporarily stores the data acquired from the camera adapter 120 on the DRAM, and buffers the foreground image, the background image, the audio data, and the three-dimensional model data until they are prepared. The foreground image, background image, audio data, and three-dimensional model data are collectively referred to as shooting data hereafter. Meta information such as routing information, time code information (time information), and camera identifier is added to the shooting data, and the front-end server 230 confirms the data attributes based on this meta information. As a result, the front-end server 230 determines that the data is at the same time and confirms that the data is complete. This is because the reception order of network packets is not guaranteed for the data transferred from each camera adapter 120 by the network, and it is necessary to buffer until the data necessary for file generation is prepared.

Further, the background image may be taken at a frame rate different from that of the foreground image. For example, when the frame rate of the background image is 1 fps, one background image is acquired every second. Therefore, for the time when the background image is not acquired, all the data may be collected without the background image. .. Further, the front-end server 230 notifies the database 250 of information indicating that the data cannot be collected if the data is not available even after the lapse of a predetermined time.

The database 250 manages the reception status of each frame and image data from each camera adapter 120 acquired from the front-end server 230 in the state management table. For example, it can be dealt with by setting a flag of 0 if the image data has not arrived for each time and each camera, and 1 if the image data has arrived. Further, it can be managed by setting a similar flag for each time within a predetermined time and for each camera if all of them have not arrived at a predetermined time (for example, 1 second).

The back-end server 270 receives the designation of the virtual viewpoint from the virtual camera operation UI 330, reads the corresponding image and audio data from the database 250 based on the accepted viewpoint, performs rendering processing, and generates the virtual viewpoint image. At this time, the database 250 provides data to the back-end server 270 in response to the read request from the back-end server 270 according to the reception status of the state management table.

The configuration of the image computing server 200 is not limited to this. For example, at least two of the front-end server 230, the database 250, and the back-end server 270 may be integrally configured. Further, at least one of the front-end server 230, the database 250, and the back-end server 270 may exist in a plurality. Further, a device other than the above device may be included at an arbitrary position of the image computing server 200. Further, the end user terminal 190 or the virtual camera operation UI 330 may have at least a part of the functions of the image computing server 200.

The rendered virtual viewpoint image is transmitted from the back-end server 270 to the end user terminal 190, and the user who operates the end user terminal 190 can view the image and listen to the sound according to the designation of the viewpoint. That is, the back-end server 270 generates virtual viewpoint contents based on the captured images (multi-viewpoint images) captured by the plurality of cameras 112 and the viewpoint information. More specifically, the back-end server 270 is virtual based on, for example, image data in a predetermined area cut out from images taken by a plurality of cameras 112 by a plurality of camera adapters 120 and a viewpoint designated by a user operation. Generate perspective content. Then, the back-end server 270 provides the generated virtual viewpoint content to the end user terminal 190.

The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, the virtual viewpoint image can be said to be an image representing the appearance at the specified viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. Further, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to the viewpoint automatically designated by the device are also included in the virtual viewpoint image. In this embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described, but the audio data may not necessarily be included.

The virtual camera operation UI 330 accesses the database 250 via the back-end server 270. The back-end server 270 performs common processing related to image generation processing, and the difference portion of the application related to the operation UI is performed by the virtual camera operation UI 330.

As described above, in the image processing system 100, the back-end server 270 generates a virtual viewpoint image based on the images captured by the plurality of cameras 112 for photographing the subject from a plurality of directions. The image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be logically configured.

<Explanation of functional block diagram>
Next, a functional block diagram of the camera adapter 120, the back-end server 270, and the virtual camera operation UI 330 in the system shown in FIG. 1 will be described.

FIG. 2 is a block diagram for explaining the functional configuration of the camera adapter 120. The camera adapter 120 includes a network adapter 6110, a transmission unit 6120, an image processing unit 6130, and an external device control unit 6140. The network adapter 6110 includes a data transmission / reception unit 6111 and a time control unit 6112.

The data transmission / reception unit 6111 performs data communication with the other camera adapter 120, the front-end server 230, the time server 290, and the control station 310 via the daisy chain 170, the network 291 and the network 310a. For example, the data transmission / reception unit 6111 outputs the foreground image and the background image separated by the foreground background separation unit 6131 from the image captured by the camera 112 to another camera adapter 120. The output destination camera adapter 120 is the next camera adapter 120 in the camera adapter 120 of the image processing system 100 in a predetermined order according to the processing of the data routing processing unit 6122. Each camera adapter 120 outputs a foreground image and a background image to generate a virtual viewpoint image based on the foreground image and the background image taken from a plurality of viewpoints. There may be a camera adapter 120 that outputs a foreground image separated from a captured image and does not output a background image.

The time control unit 6112 has, for example, a function of storing a time stamp of data transmitted / received to / from the time server 290 and a function of performing time synchronization with the time server 290 in accordance with the Oldinay Clock of the IEEE 1588 standard. It is not limited to IEEE1588, and time synchronization with a time server may be realized by another EtherAVB standard or an original protocol.

The transmission unit 6120 includes a data compression / decompression unit 6121, a data routing processing unit 6122, a time synchronization control unit 6123, an image / audio transmission processing unit 6124, a data routing information holding unit 6125, and a frame rate changing unit 6126. including.

The data compression / decompression unit 6121 has a function of compressing data transmitted / received via the data transmission / reception unit 6111 by applying a predetermined compression method, compression rate, and frame rate, and a function of decompressing the compressed data. And have.

The data routing processing unit 6122 uses the data held by the data routing information holding unit 6125, which will be described later, to determine the routing destination of the data received by the data transmitting / receiving unit 6111 and the data processed by the image processing unit 6130. Further, it has a function of transmitting data to the determined routing destination. As the routing destination, a camera adapter 120 corresponding to the cameras 112 focused on the same gazing point is suitable for performing image processing because the image frame correlation between the cameras 112 is high. The order of the camera adapters 120 that output the foreground image and the background image in the relay format in the image processing system 100 is determined according to the determination by the data routing processing unit 6122 of each of the plurality of camera adapters 120.

The time synchronization control unit 6123 complies with PTP (Precision Time Protocol) of the IEEE1588 standard, and has a function of performing processing related to time synchronization with the time server 290. It should be noted that the time is not limited to PTP, and time synchronization may be performed using another similar protocol.

The image / audio transmission processing unit 6124 has a function of creating a message for transferring image data or audio data to another camera adapter 120 or a front-end server 230 via the data transmission / reception unit 6111. The message includes image data or audio data, and meta information of each data. The meta information of the present embodiment includes a time code or sequence number at the time of image capture or sound sampling, a data type, an identifier indicating an individual of the camera 112 or the microphone 111, and the like. The image data or audio data to be transmitted may be data-compressed by the data compression / decompression unit 6121. Further, the image / voice transmission processing unit 6124 receives a message from another camera adapter 120 via the data transmission / reception unit 6111. Then, depending on the data type included in the message, it is fragmented into the packet size specified by the transmission protocol and restored to image data or voice data. If the data is compressed when the data is restored, the data compression / decompression unit 6121 performs decompression processing.

The data routing information holding unit 6125 has a function of holding address information for determining a transmission destination of data transmitted / received by the data transmitting / receiving unit 6111. The data routing information holding unit 6125 holds, for example, the installation information of a plurality of cameras 112 focused on the same gazing point and the adjacent camera identifiers based on the installation information. The installation information is transmitted from the control station 310 to each camera adapter 120 and stored in the data routing information holding unit 6125. For example, in FIG. 1, in a configuration in which data is transmitted between camera adapters 120 in a daisy chain connection, the transmission order may be determined in advance. Specifically, based on the installation information and transmission order information transmitted from the control station 310 so that the two cameras 112 located far from the switching hub 180 transmit in the switching hub 180 direction in different directions. , Determine the transmission route of image data. With this configuration, it is possible to transmit the image-processed data to yet another camera adapter 120 by using the data correlation between the camera adapters 120.

The frame rate changing unit 6126 has a function of switching the frame rate for transferring image data to another camera adapter 120 or the front end server 230 via the data transmitting / receiving unit 6111. Further, the frame rate changing unit 6126 also has a function of receiving a frame rate changing message transmitted from the back-end server 270. The frame rate change message includes a camera identifier and meta information indicating the type of frame rate indicating whether it is a normal frame rate or a high frame rate. The process of switching the frame rate of the image data is executed in response to the frame rate change message. By this switching process, either a normal frame rate sufficient for a live image (for example, 60 frames per second) or a high frame rate for a replay image capable of slow playback (for example, 240 frames per second) is set. The frame rate is not limited to two stages, and may be appropriately set according to the amount of data transmission such as resolution and the image specifications.

The image / audio transmission processing unit 6124 holds the frame data of the image data at the maximum frame rate, and extracts and transmits the data of the corresponding frame at the normal frame rate based on the frame rate change message. ..

Further, the camera control unit 6141 may be configured to switch the output frame rate with the camera 112 according to the frame rate change message as appropriate. In this case, the memory that temporarily holds the image data can be reduced, but considering the transmission speed, it is temporarily stored at the maximum frame rate, and the frame rate is changed according to the frame rate change message. It is preferable to change.

The image processing unit 6130 includes a foreground background separation unit 6131, a three-dimensional model information generation unit 6132, and a calibration control unit 6133. The image processing unit 6130 processes the image data taken by the camera 112 and the image data received from the other camera adapter 120 under the control of the camera control unit 6141.

The foreground background separating unit 6131 has a function of separating the image data taken by the camera 112 into a foreground image and a background image. That is, the foreground background separation unit 6131 of each of the plurality of camera adapters 120 extracts a predetermined area from the image captured by the corresponding camera 112 among the plurality of cameras 112. The predetermined area is, for example, an area in which an object is detected as a result of an object detection process for a captured image. The foreground background separation unit 6131 uses the image in the predetermined area extracted from the captured image as the foreground image, the image in the area other than the predetermined area (that is, other than the foreground image) as the background image, and the captured image as the foreground image and the background image. To separate. The object is, for example, a person. However, the object is not limited to this, and the object may be a specific person (player, manager, and / or referee, etc.) or an object having a predetermined image pattern such as a ball. Further, a moving object may be detected as an object. The image of the part corresponding to the above object of the virtual viewpoint image generated in the image processing system 100 by separately processing the foreground image including an important object such as a person and the background image not including such an object. The quality of the can be improved. Further, by separating the foreground image and the background image by each of the plurality of camera adapters 120, it is possible to distribute the load in the image processing system 100 provided with the plurality of cameras 112.

The 3D model information generation unit 6132 uses the foreground image separated by the foreground background separation unit 6131 and the foreground image received from another camera adapter 120, and uses, for example, an image related to the 3D model using the principle of a stereo camera. It has a function to generate information.

The calibration control unit 6133 performs color correction processing for suppressing color variation for each camera for the input image, image alignment processing and image cropping processing for blurring caused by camera vibration, and the like. I do.

The calibration control unit 6133 has a function of acquiring image data necessary for calibration from the camera 112 via the camera control unit 6141 and transmitting it to the front-end server 230 that performs arithmetic processing related to calibration. .. In the present embodiment, the arithmetic processing related to the calibration is performed by the front-end server 230, but the node that performs the arithmetic processing is not limited to the front-end server 230. For example, arithmetic processing may be performed at another node such as the control station 310 or the camera adapter 120 (including another camera adapter 120). Further, the calibration control unit 6133 has a function of performing calibration (dynamic calibration) during shooting according to preset parameters for the image data acquired from the camera 112 via the camera control unit 6141. Have.

The external device control unit 6140 has a function of controlling a device connected to the camera adapter 120, and includes a camera control unit 6141, a microphone control unit 6142, a pan head control unit 6143, and a sensor control unit 6144.

The camera control unit 6141 has a function of connecting to the camera 112 and performing control of the camera 112, acquisition of captured images, provision of synchronization signals, time setting, and the like. The control of the camera 112 includes, for example, setting and reference of shooting parameters (number of pixels, color depth, frame rate, white balance, etc.), and the state of the camera 112 (shooting, stopped, synchronizing, error, etc.). There are acquisition, start and stop of shooting, focus adjustment, etc. The synchronization signal is provided by the time synchronization control unit 6123 using the time synchronized with the time server 290 and providing the shooting timing (control clock) to the camera 112. The time setting is performed by providing the time synchronized with the time server 290 by the time synchronization control unit 6123 with a time code conforming to the format of, for example, SMPTE12M. As a result, the provided time code is added to the image data received from the camera 112.

The microphone control unit 6142, the sensor control unit 6144, and the pan head control unit 6143 have a function of controlling the microphone 111, the external sensor 114, and the pan head 113, respectively, which are connected to each other.

<Explanation of backend server 270>
FIG. 3 is a block diagram for explaining the functional configuration of the back-end server 270. The back-end server 270 includes a data receiving unit 3001, a background texture pasting unit 3002, a foreground texture determining unit 3003, a foreground texture boundary color matching unit 3004, a virtual viewpoint foreground image generation unit 3005, and a rendering unit 3006. Have. Further, it has a virtual viewpoint audio generation unit 3007, a synthesis unit 3008, a virtual viewpoint content output unit 3009, a foreground object determination unit 3010, a request list generation unit 3011, and a request data output unit 3012. It also has a background mesh model management unit 3013, a rendering mode management unit 3014, and a camera selection unit 3015.

The data receiving unit 3001 receives the data transmitted from the database 250 and the controller 300. From the database 250, three-dimensional data (background mesh model) showing the shape of the stadium, a foreground image, a background image, a three-dimensional model of the foreground image (hereinafter referred to as a foreground three-dimensional model), and audio are received.

Further, the data receiving unit 3001 receives the virtual camera parameters output from the controller 300 that specifies the viewpoint related to the generation of the virtual viewpoint image. The virtual camera parameter is data representing the position and posture of the virtual viewpoint, and for example, a matrix of external parameters and a matrix of internal parameters are used.

Further, the data receiving unit 3001 may acquire information about a plurality of cameras 112 from an external device such as a database 250 or a controller 300. The information regarding the plurality of cameras 112 is, for example, information regarding the number of the plurality of cameras 112, information regarding the operating state of the plurality of cameras 112, and the like. The operating state of the camera 112 includes, for example, at least one of a normal state, a failure state, a standby state, a starting state, and a restarting state of the camera 112. In addition, information indicating that data collection is not possible at the same time is also acquired because the data output from each camera adapter 120 has different frame rates.

The background texture pasting unit 3002 pastes the background image as a texture for the three-dimensional spatial shape indicated by the background mesh model acquired from the background mesh model management unit 3013. Doing so will generate a textured background mesh model. The mesh model is data that represents a three-dimensional spatial shape as a set of faces, such as CAD data. A texture is an image that is pasted to express the texture of the surface of an object.

The foreground texture determination unit 3003 determines the texture information of the foreground 3D model from the foreground image and the foreground 3D model group. The foreground texture boundary color matching unit 3004 performs color matching of the texture boundary from each foreground 3D model and each 3D model group, and generates a colored foreground 3D model group for each foreground object.

The virtual viewpoint foreground image generation unit 3005 performs fluoroscopic transformation of the foreground image group so that it looks from the virtual viewpoint based on the virtual camera parameters. The rendering unit 3006 renders the background image and the foreground image based on the generation method used for generating the virtual viewpoint image determined by the rendering mode management unit 3014 to generate the virtual viewpoint image of the whole view.

In this embodiment, two rendering modes, model-based rendering (MBR) and image-based (Image-Based Rendering: IBR), are used as a virtual viewpoint image generation method.

The MBR is a method of generating a virtual viewpoint image using a three-dimensional model generated based on a plurality of captured images of a subject captured from a plurality of directions. When the rendering mode is MBR, a panoramic model is generated by synthesizing the background mesh model and the foreground three-dimensional model group generated by the foreground texture boundary color matching unit 3004, and a virtual viewpoint image is generated from the panoramic model.

IBR is a technique for generating a virtual viewpoint image that reproduces the appearance from a virtual viewpoint by transforming and synthesizing a group of input images obtained by capturing a target scene from a plurality of viewpoints. In this embodiment, when IBR is used, a virtual viewpoint image is generated based on one or a plurality of captured images, which is smaller than a plurality of captured images for generating a three-dimensional model using the MBR. When the rendering mode is IBR, a background image viewed from a virtual viewpoint is generated based on the background texture model, and a virtual viewpoint image is generated by synthesizing the foreground image generated by the virtual viewpoint foreground image generation unit 3005. To. The rendering unit 3006 may use a rendering method other than MBR and IBR.

The rendering mode management unit 3014 determines the rendering mode as the generation method used for generating the virtual viewpoint image, and holds the determination result. In the present embodiment, the rendering mode management unit 3014 determines the rendering mode to be used from the plurality of rendering modes. This determination is made based on the information acquired by the data receiving unit 3001. For example, the rendering mode management unit 3014 determines in IBR the generation method used for generating the virtual viewpoint image when the number of cameras specified from the acquired information is equal to or less than the threshold value.

On the other hand, when the number of cameras is larger than the threshold value, the generation method is determined to be MBR. As a result, when the number of cameras is large, the range in which the viewpoint can be specified becomes wide by generating a virtual viewpoint image using the MBR. Further, when the number of cameras is small, by using the IBR, it is possible to avoid the deterioration of the image quality of the virtual viewpoint image due to the deterioration of the accuracy of the three-dimensional model when the MBR is used. Further, for example, the generation method may be determined based on the length of the allowable processing delay time from shooting to image output. MBR is used when the degree of freedom of the viewpoint is prioritized even if the delay time is long, and IBR is used when the delay time is required to be short. Further, for example, when the data receiving unit 3001 acquires information indicating that the height of the virtual viewpoint can be specified by the controller 300 or the end user terminal 190, the generation method used for generating the virtual viewpoint image is MBR. To decide. As a result, it is possible to prevent the user's request to change the height of the virtual viewpoint from being unacceptable due to the generation method being IBR. In this way, by determining the generation method of the virtual viewpoint image according to the situation, the virtual viewpoint image can be generated by the appropriately determined generation method. Further, by configuring the plurality of rendering modes to be switchable according to the request, the system can be flexibly configured, and the present embodiment can be applied to a subject other than the stadium.

Further, when the information indicating that the data cannot be collected exists instead of the image data, it is possible to prevent the image from being lost by performing the processing by interpolating from the surrounding images.

The virtual viewpoint sound generation unit 3007 generates sound (voice group) that can be heard in the virtual viewpoint based on the virtual camera parameters. The compositing unit 3008 synthesizes the image group generated by the rendering unit 3006 and the voice generated by the virtual viewpoint voice generation unit 3007 to generate virtual viewpoint content.

The virtual viewpoint content output unit 3009 outputs virtual viewpoint content to the controller 300 and the end user terminal 190 using Ethernet (registered trademark). However, the transmission means to the outside is not limited to Ethernet (registered trademark), and signal transmission means such as SDI, DisplayPort, and HDMI (registered trademark) may be used. The back-end server 270 may output a virtual viewpoint image that does not include audio and is generated by the rendering unit 3006.

The foreground object determination unit 3010 determines the foreground object group to be displayed from the virtual camera parameters and the position information of the foreground object indicating the position of the foreground object in space included in the foreground 3D model, and outputs the foreground object list. do. That is, the foreground object determination unit 3010 performs a process of mapping the image information of the virtual viewpoint to the physical camera 112.

The request list generation unit 3011 generates a request list for requesting the foreground image group and the foreground three-dimensional model group corresponding to the foreground object list at the specified time, and the background image and the audio data from the database 250. For the foreground object, the data selected in consideration of the virtual viewpoint is requested from the database 250, but for the background image and the audio data, all the data related to the frame is requested. After starting the backend server 270, a request list of the background mesh model is generated until the background mesh model is acquired.

The request data output unit 3012 outputs a data request command to the database 250 based on the input request list. The background mesh model management unit 3013 stores the background mesh model received from the database 250.

The camera selection unit 3015 makes a determination for setting the frame rate of the camera and selects a camera identifier for setting the frame rate from the background image and the foreground object included in the foreground three-dimensional model. In addition, a frame rate change message for transmission to the frame rate change unit 6126 in the camera adapter 120 is generated and transmitted. The frame rate change message includes a camera identifier and meta information indicating the type of frame rate, which is normal frame rate or high frame rate.

FIG. 4 is a diagram for explaining the operation of the camera selection unit 3015. In FIG. 4, a plurality of cameras 9100a, ..., 9100g for photographing a subject from different directions are installed. The camera selection unit 3015 according to the present embodiment selects, for example, a camera having a field of view capable of determining the line crossing of a ball game as a camera for setting a high frame rate. That is, the camera selection unit 3015 serves as a camera that sets a camera having a predetermined positional relationship with the region of interest at a high frame rate when the object of interest (hereinafter referred to as “object of interest”) enters the region of interest. Select. For example, the shooting target is a soccer game. When a ball, which is an object of interest, enters a region near the field line (area of interest), a camera located near the vicinity region, for example, a straight line of the field line. Select the camera located above as the high frame rate camera.

Specifically, a case where the object of interest 9101 (ball in this example) moves from the point 9101a to the point 9101b to the point 9101c and enters the vicinity region 9103 of the field line 9102 will be described with reference to FIG. In this case, the camera selection unit 3015 selects the camera 9100f closest to the extended straight line of the field line 9102 among the plurality of cameras 9100. In addition, in order to generate a virtual viewpoint image with higher image quality, the adjacent cameras 9100e and 9100g may also be selected. That is, the camera selection unit 3015 selects the camera in the selection range 9104 according to the fact that the object of interest 9101 enters the neighborhood region 9103 of the field line 9102.

The timing of camera selection is when the object enters the area of interest. For example, the neighborhood area 9103 of the field line 9102 in the background image is set in advance as a region of interest. Then, the camera selection unit 3015 compares the coordinate values on the world coordinate system of the object of interest with the coordinate values of the region of interest, and when it is determined that the object of interest has entered the region of interest, the camera selection process is executed.

The area of interest is appropriately set based on the scene or place of interest in the shooting target. For example, in the case of soccer, it is set based on the distance information from the line or goal post, and in the case of baseball, it is set based on the vicinity of each base, the mound, the pole, and the like. By setting in this way, it is possible to use the generated replay image for determining a sports competition, or to slow-play a range in which important play is likely to be performed.

FIG. 5 is a diagram for explaining the selected camera. In FIG. 5, among the cameras 9200a, ..., 9200z, 9200A, ..., 9200D installed around the field 9205 of the stadium, the black-painted camera has a high frame rate, and the white camera has a normal frame. It's a rate camera.

The camera selection unit 3015 selects a camera having a field of view capable of determining whether or not the object of interest 9201 has crossed the field line 9202. That is, when the object of interest 9201 enters the region near the field line 9202, a camera installed at a position close to the straight line 9203 of the field line 9202 is selected. Specifically, the cameras 9200i and 9200j shoot the object of interest 9201 from the first direction, and the cameras 9200p and 9200q shoot from the second direction facing the first direction. The number of selected cameras may be appropriately set according to the situation.

In addition, the camera selection unit 3015 rotates the straight line 9203 by 90 degrees with respect to the coordinate position of the object of interest 9201, that is, at a position close to the straight line 9204 orthogonal to the position of the straight line 9203 and the object of interest 9201. Select the installed camera. Specifically, the cameras are 9200m and 9200n, and the cameras are 9200A and 9200B. This is to meet the demand for viewing from various directions when viewing the virtual viewpoint image as a replay image.

Further, the camera selection unit 3015 selects other cameras, for example, alternately, that is, every other camera with a high frame rate. Specifically, the cameras are 9200D, 9200b, 9200d, 9200f, 9200h, 9200l, 9200r, 9200t, 9200v, 9200x, 9200z. This is to generate a virtual viewpoint image as described above, and to generate a three-dimensional model based on a plurality of captured images captured from a plurality of directions by the above-mentioned MBR.

The camera selection unit 3015 generates a frame rate change message with the frame rate as the high frame rate for the finally selected camera. It also generates a frame rate change message with the frame rate as the normal frame rate for the unselected cameras. The generated frame rate change message is input to each sensor system 110 via the network 270a. The camera adapter 120 of each sensor system 110 receives this and inputs it to the frame rate changing unit 6126 (see FIG. 2), and the frame rate changing unit 6126 changes the frame rate according to the content of the message.

At the time of the image data taken by the camera set to the normal frame rate, the image data taken by the camera set to the high frame rate at the same time exists in the database 250. That is, image data at the same time taken by all cameras having a normal frame rate and a high frame rate are collected in the database 250. However, in the frame other than the time taken by the camera set to the normal frame rate, only the image data of the camera set to the high frame rate exists in the database 250. Therefore, the image data of all the cameras cannot be collected in the database 250 except the time when the camera set to the normal frame rate shoots.

When the image data is collected in the database 250, the back-end server 270 can perform rendering processing to generate a virtual viewpoint image as described above. However, if the image data at the time in the database 250 cannot be collected, the back-end server 270 cannot generate the virtual viewpoint image.

Therefore, in the present embodiment, as described above, the camera selection unit 3015 selects at least every other camera as a camera having a normal frame rate and a high frame rate. That is, the image data of at least one of the adjacent cameras is set to exist at the same time. Therefore, the image data of the camera having no image data at the time can generate the image data of the camera from the image data of the adjacent camera.

In the virtual viewpoint foreground image generation unit 3005, for example, a projective transformation is used to generate the image data. Since the projective transformation is a general image process, details will not be described here, but it is a transformation that projects onto the plane of the camera by using the image data of the plane of the adjacent camera. By this conversion, it is possible to generate image data of the camera without frame data. Further, new pixel data may be calculated from the motion information of the pixel data of the image data of the normal frame rate before and after the time to be generated, and the image data may be generated.

The execution of each process when the above-mentioned image data can be collected and when the image data cannot be collected is based on the data collection state of the state management table held by the database 250 read by the back-end server 270. By the way, in the above configuration, it is assumed that the cameras are set to the normal frame rate and the high frame rate at least every other camera. However, the configuration is not limited to this, and it is not always necessary to use every other unit if it is possible to generate image data by projective transformation when the image data cannot be collected.

FIG. 6 is a diagram for explaining the virtual camera 8001. In FIG. 6A, the virtual camera 8001 is a virtual camera capable of taking a picture from a viewpoint different from that of any installed camera 112. That is, the virtual viewpoint image generated by the image processing system 100 is an image taken by the virtual camera 8001. In FIG. 6A, each of the plurality of sensor systems 110 arranged on the circumference has a camera 112. For example, by generating a virtual viewpoint image, it is possible to generate an image as if it was taken by a virtual camera 8001 near a soccer goal. The virtual viewpoint image, which is an image captured by the virtual camera 8001, is generated by image processing the images of a plurality of installed cameras 112. The operator (user) can obtain a captured image from a free viewpoint (line-of-sight direction) by operating the position of the virtual camera 8001 and the like. In FIG. 6B, the virtual camera path 8002 shows a sequence of information representing the position and posture of the virtual camera 8001 for each frame. Details will be described later.

FIG. 7 is a block diagram for explaining the functional configuration of the virtual camera operation UI 330. The virtual camera operation UI 330 includes a virtual camera management unit 8130 and an operation UI unit 8120. These may be mounted on the same device, or may be mounted separately on the server device and the client device, respectively.

The virtual camera operation unit 8101 receives an operation on the virtual camera 8001 of the operator, that is, an instruction by the user for designating a viewpoint related to the generation of the virtual viewpoint image. The operator's operation content is, for example, a change in the position (movement) of the virtual camera 8001, a change in the posture (rotation), a change in the zoom magnification, and the like. Further, the virtual camera operation unit 8101 is used to generate a live image and a replay image. When generating a replay image, an operation of specifying a time in addition to the position and posture of the camera is performed. In the replay image, for example, it is possible to stop the time and move the virtual camera 8001.

The virtual camera parameter derivation unit 8102 derives virtual camera parameters representing the position, orientation, and the like of the virtual camera 8001. The virtual parameter may be derived by an operation, or may be derived by a reference to a look-up table or the like. As the virtual camera parameters, for example, a matrix representing an external parameter and a matrix representing an internal parameter are used. Here, the position and orientation of the virtual camera 8001 are included in the external parameters, and the zoom value is included in the internal parameters.

The virtual camera constraint management unit 8103 acquires and manages information for specifying a restricted area in which the designation of the viewpoint based on the instruction received by the virtual camera operation unit 8101 is restricted. This information is, for example, a constraint regarding the position, posture, zoom value, etc. of the virtual camera 8001.

The collision determination unit 8104 determines whether the virtual camera parameter derived by the virtual camera parameter derivation unit 8102 satisfies the virtual camera constraint. If the constraint is not satisfied, for example, the operation input by the operator is canceled, the virtual camera 8001 is controlled not to move from the position satisfying the constraint, or the virtual camera 8001 is returned to the position satisfying the constraint.

The feedback output unit 8105 feeds back the determination result of the collision determination unit 8104 to the operator. For example, when the virtual camera constraint is not satisfied by the operator's operation, the operator is notified.

The virtual camera path management unit 8106 manages the path (virtual camera path 8002) of the virtual camera 8001 according to the operation of the operator. As shown in FIG. 6B, the virtual camera path 8002 is a sequence of information representing the position and orientation of the virtual camera 8001 for each frame. For example, virtual camera parameters are used as information representing the position and posture of the virtual camera 8001. For example, the information for one second in the setting of the frame rate of 60 frames / second is a sequence of 60 virtual camera parameters. The virtual camera path management unit 8106 transmits the virtual camera parameters determined by the collision determination unit 8104 to the back-end server 270. The back-end server 270 uses the received virtual camera parameters to generate a virtual viewpoint image and a virtual viewpoint sound. Further, the virtual camera path management unit 8106 also has a function of adding and holding virtual camera parameters to the virtual camera path 8002. For example, when the virtual camera operation UI 330 is used to generate the virtual viewpoint image and the virtual viewpoint sound for one hour, the virtual camera parameters for one hour are saved as the virtual camera path 8002.

The authoring unit 8107 provides an editing function when the operator generates a replay image. The authoring unit 8107 takes out a part of the virtual camera path 8002 held by the virtual camera path management unit 8106 as the initial value of the virtual camera path 8002 for the replay image according to the user operation. The virtual camera image / sound output unit 8108 outputs the virtual camera image / sound received from the back-end server 270. The operator operates the virtual camera 8001 while checking the output image and sound.

<Explanation of processing of camera selection unit 3015>
Subsequently, the processing of the camera selection unit 3015 of the back-end server 270 will be described. FIG. 8 is a flowchart showing the processing of the camera selection unit 3015.

In step S101, the camera selection unit 3015 determines whether or not there is an object of interest in the foreground image. This process executes determination of whether or not the foreground image contains an object of interest (for example, a ball) that is appropriately set according to the shooting scene. This determination is performed, for example, by extracting a feature amount from image data and collating it with the learning result data obtained by a learning algorithm to obtain a recognition result. Further, the object of interest may be detected by a predetermined image process such as motion detection or image recognition. The present invention is not limited to this, and the user may manually specify it from the control station 310 or the like. If it is determined that the foreground image has an object of interest (YES in step S101), the process proceeds to step S102. If not (NO in step S101), the process proceeds to step S110.

In step S102, the camera selection unit 3015 determines whether or not there is a line (attention line) of interest in the background image. This process is based on the background image data output from the database 250. The attention line is set in advance by the user using the end user terminal 190. Performs a process of confirming whether the background image data includes a range of interest, such as a field line, which is appropriately set according to the shooting scene. If it is determined that the background image has a line of interest (YES in step S103), the process proceeds to step S103. If not (NO in step S102), the process proceeds to step S110. The line of interest may be, for example, a line on the field where the competition is performed. Further, the line of interest may be a straight line passing through the position of the object of interest.

In step S103, the camera selection unit 3015 executes a process of comparing the coordinate positions of the object of interest and the line of interest. In this process, the comparison process is performed using the coordinate information on the world coordinate system such as the object and the line extracted in step S101 and step S102, and the comparison process result such as the difference information is temporarily saved.

In step S104, the camera selection unit 3015 determines whether or not the object of interest exists in the region of interest. For example, in FIG. 5, the camera selection unit 3015 determines whether or not the object of interest 9201 exists in the vicinity of the field line 9202. If it is determined that the object of interest is within the region of interest (YES in step S104), the process proceeds to step S105. If not (NO in step S104), the process proceeds to step S110. The area of interest may be, for example, a center area on the field where the competition is played, a penalty area, a vicinity of a kick mark, a vicinity of a side line, a vicinity of a goal line, or a goal area.

In step S105, the camera selection unit 3015 selects a camera located near the extension line of the line of interest based on the coordinate value data of the line of interest acquired in step S102. For example, in FIG. 5, the cameras 9200i, 9200j and the cameras 9200q, 9200p located near the straight line 9203 of the field line 9202 are selected. The camera installation information is positioned in advance as the coordinate values of the world coordinate system, and the target camera is specified by appropriately reading the installation information and projecting and transforming it into a plane coordinate system.

In step S106, the camera selection unit 3015 rotates the coordinates of the attention line acquired in step S102 by 90 degrees with respect to the coordinate position of the attention object, and selects a camera at a position close to the extended straight line of the rotated attention line. .. For example, in FIG. 5, cameras 9200A and 9200B and cameras 9200m and 9200n, which are close to the straight line 9204 obtained by rotating the straight line 9203 by 90 degrees, are selected. The coordinates of the line of interest acquired in step S102 may be rotated at an arbitrary angle instead of rotating at 90 degrees with respect to the coordinate position of the object of interest. Further, the coordinates of the line of interest may be rotated in an angle band having a width such as 60 degrees to 120 degrees with respect to the coordinate position of the object of interest.

In step S107, the camera selection unit 3015 executes, for example, a process of selecting every other camera for the cameras other than the cameras selected in steps S105 and S106. In step S108, the camera selection unit 3015 creates a frame rate change message including information for changing the frame rate for the selected camera.

In step S109, the camera selection unit 3015 transmits a frame rate change message to the frame rate change unit 6126 of the camera adapter 120, and ends the process. In step S110, the camera selection unit 3015 generates meta information based on the determination result, outputs the meta information to the end user terminal 190, and ends the process.

By performing the above-mentioned series of processing, in an image processing system including a plurality of cameras, it is possible to generate and output a message that selectively sets the camera that shoots the object of interest to a high frame rate.

<Explanation of processing of virtual viewpoint foreground image generation unit 3005>
FIG. 9 is a flowchart showing the processing of the virtual viewpoint foreground image generation unit 3005 of the back-end server 270. In this system, since cameras with a normal frame rate and a high frame rate are set, image data may or may not be collected in the database 250 at a certain time. The virtual viewpoint foreground image generation unit 3005 performs a process of generating image data when the image data cannot be collected. After the processing of the virtual viewpoint foreground image generation unit 3005, the rendering unit 3006 generates the virtual viewpoint image.

In step S201, the virtual viewpoint foreground image generation unit 3005 reads out the state management table indicating the data collection state from the database 250. In step S202, the virtual viewpoint foreground image generation unit 3005 reads out whether or not all the image data of the target camera necessary for virtual viewpoint image generation exists at a certain time based on the virtual camera parameter. Judge from. When it is determined that all the image data of the target camera exists (YES in step S202), the process ends. If it is determined that some of the image data of the target camera does not exist (NO in step S202), the process proceeds to step S203.

In step S203, the virtual viewpoint foreground image generation unit 3005 sets, for example, the camera identifier of the target camera with the smallest numerical value as an initial value. In step S204, the virtual viewpoint foreground image generation unit 3005 determines whether or not there is image data of the set camera identifier based on the state management table. If it is determined that there is no image data (NO in step S204), the process proceeds to step S205. If it is determined that there is image data (YES in step S204), the process proceeds to step S207.

In step S205, the virtual viewpoint foreground image generation unit 3005 acquires data for use in image interpolation from the database 250 in order to generate image data. Here, the data acquired from the database 250 is, for example, image data of a camera adjacent to the camera.

In step S206, the virtual viewpoint foreground image generation unit 3005 executes a process of generating an interpolated image of the target camera by performing a projective transformation of the image data acquired in step S205.

In step S207, the virtual viewpoint foreground image generation unit 3005 determines whether or not the check for the existence of all the image data of the target camera has been completed. If it is determined that the check has not been completed (NO in step S207), the process returns to step S204, and among the camera identifiers of the target cameras, the one with the smallest numerical value next to the current camera identifier is checked. On the other hand, when it is determined that the check for whether or not all the image data of the target camera exists (YES in step S207), the virtual viewpoint foreground image generation unit 3005 ends the process.

In the present embodiment, the number of high frame rate cameras is determined in advance by the total number of cameras and the data capacity, but the configuration is not limited to this. For example, a bandwidth monitoring unit may be configured in the front-end server 230. In this case, the band monitoring unit calculates the number of cameras according to the data capacity for transmitting the band, and feedback control is performed to the camera selection unit 3015.

<About hardware configuration>
Subsequently, the hardware configuration of each device constituting the present embodiment will be described. As described above, in the present embodiment, an example in which the camera adapter 120 implements hardware such as FPGA and / or ASIC and executes each of the above-mentioned processes by these hardware has been mainly described. The same applies to the various devices in the sensor system 110, the front-end server 230, the database 250, the back-end server 270, and the controller 300. However, at least one of the above devices may use, for example, a CPU, GPU, DSP, or the like to execute the processing of the present embodiment by software processing.

FIG. 10 is a block diagram showing a hardware configuration of the camera adapter 120. Devices such as the front-end server 230, the database 250, the back-end server 270, the control station 310, the virtual camera operation UI 330, and the end-user terminal 190 may also have the hardware configuration shown in FIG. The camera adapter 120 includes a CPU 1201, a ROM 1202, a RAM 1203, an auxiliary storage device 1204, a display unit 1205, an operation unit 1206, a communication unit 1207, and a bus 1208.

The CPU 1201 controls the entire camera adapter 120 by using computer programs and data stored in the ROM 1202 and the RAM 1203. ROM 1202 stores programs and parameters that do not need to be changed. The RAM 1203 temporarily stores programs and data supplied from the auxiliary storage device 1204, data supplied from the outside via the communication unit 1207, and the like. The auxiliary storage device 1204 is composed of, for example, a hard disk drive or the like, and stores content data such as still images and moving images.

The display unit 1205 is composed of, for example, a liquid crystal display or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the camera adapter 120. The operation unit 1206 is composed of, for example, a keyboard, a mouse, or the like, and inputs various instructions to the CPU 1201 in response to an operation by the user. The communication unit 1207 communicates with an external device such as a camera 112 or a front-end server 230. Bus 1208 connects each part of the camera adapter 120 to transmit information.

The above-described embodiment has mainly described an example in which the image processing system 100 is installed in a facility such as a stadium or a concert hall. Other examples of facilities include, for example, amusement parks, parks, racetracks, keirin racetracks, casinos, pools, skating rinks, ski resorts, live houses, and the like. In addition, the events held at various facilities may be held indoors or outdoors. In addition, the facilities in this embodiment include facilities that are temporarily constructed (for a limited time).

<Embodiment 2>
In the first embodiment, the camera selection unit selects a camera having a high frame rate based on a predetermined procedure when the object enters a predetermined range of the imaging region. The present embodiment differs from the first embodiment in that a camera having a high frame rate is selected based on the positional relationship between the object and the virtual viewpoint.

FIG. 11 is a block diagram for explaining the functional configuration of the back-end server 270 according to the second embodiment. Since the same reference numerals as those in FIG. 3 explaining the back-end server 270 according to the first embodiment are as described above, detailed description thereof will be omitted. The camera selection unit 3016 according to the present embodiment is different from the camera selection unit 3015 according to the first embodiment in that the virtual camera parameters are acquired from the controller 300. The camera selection unit 3016 uses this information to make a determination for setting the frame rate of the camera, select a camera identifier, and generate and transmit a frame rate change message.

FIG. 12 is a diagram for explaining the operation of the camera selection unit 3016. 12 (a) shows cameras A to G and virtual viewpoints 9302 that shoot the object of interest 9301, and FIG. 12 (b) shows the transition of the shooting timing of the cameras A to G.

The camera selection unit 3016 in the present embodiment selects a camera whose line-of-sight direction and shooting direction of the virtual viewpoint are close to each other as a camera for setting a high frame rate. For example, the shooting target is a soccer game. When the virtual viewpoint is moving in conjunction with the ball, which is the object of interest, a camera whose line-of-sight direction and shooting direction of the virtual viewpoint are close to each other, that is, a camera located on a straight line passing through the object of interest and the virtual viewpoint, is a high frame. Select as a rate camera.

A case where the object of interest 9301 is moving in the order of the point 9301a → the point 9301b → the point 9301c will be specifically described with reference to FIG. 12A. In conjunction with the movement of the object of interest 9301, the virtual viewpoint 9302 moves in the order of point 9302a → point 9302b → point 9302c. Here, the camera B closest to the straight line passing through the object of interest 9301 and the virtual viewpoint 9302 on the world coordinate system is selected. In addition, in order to generate a virtual viewpoint image with higher image quality, the adjacent cameras A and C may also be selected. The camera selection unit 3016 selects a camera in the selection range 9303a → the selection range 9303bE) → the selection range 9303c in conjunction with the movement of the object of interest 9301 and the virtual viewpoint 9302.

The transition of the shooting timing of the cameras A to G described with reference to FIG. 12A will be described with reference to FIG. 12B. In FIG. 12B, the shooting timing for each camera is plotted. The black circles in the figure are cameras that shoot at that time, and the white circles are cameras that do not shoot at that time. The time axis information of the normal frame rate is tNn, and the time axis information of the high frame rate is tHn. A virtual viewpoint image of the object of interest is generated as a live image based on the captured image taken at the timings tN0, tN1, tN2, .... A virtual viewpoint image of the object is generated as a replay image based on the captured image taken at the timing of time tH0, tH1, tH2, ....

The timing of camera selection is the timing of the normal frame rate. For example, from the virtual viewpoint acquired at time tN0 and the position of the object of interest, the camera with the selection range 9303a is selected as the high frame rate camera at time tH0, tH1, tH2, tH3. The ranges 9304a, 9304b, and 9304c correspond to the high frame cameras selected at the times tN0, tN1, and tN2, respectively. However, in this example, the delay time from the determination of the camera to be selected to the setting of the frame rate is not taken into consideration.

Further, as described above, the camera selection unit 3016 selects cameras other than the cameras selected based on the line-of-sight direction of the virtual viewpoint with respect to the object of interest, for example, every other camera as a high frame rate camera. Specifically, the cameras A, C, E, and G. This is to generate a virtual viewpoint image, and the above-mentioned MBR generates a three-dimensional model based on a plurality of captured images captured from a plurality of directions. Image data with a high frame rate that does not exist at a certain time may be generated by interpolation using image data of adjacent cameras. As described above, the interpolated image generation process is executed by the virtual viewpoint foreground image generation unit 3005 of the back-end server 270.

FIG. 13 is a diagram for explaining the selected camera. In FIG. 13, among the cameras 9200a, ..., 9200z, 9200A, ..., 9200D installed around the field 9205 of the stadium, the black-painted camera is the high frame rate camera and the white camera is the normal frame. It's a rate camera.

The camera selection unit 3016 selects a camera close to the straight line 9403 passing through the object of interest 9402 and the virtual viewpoint 9401 so that the virtual viewpoint content can be viewed from the same line of sight as the virtual viewpoint and from the opposite direction. Specifically, it is a camera 9200u and a camera 9200j.

In addition, a camera close to the straight line 9404 obtained by rotating the straight line 9403 by 90 degrees with respect to the coordinate position of the object of interest 9402 is selected. Specifically, the cameras 9200c and 9200p.

Further, the camera selection unit 3016 selects other cameras, for example, every other camera. Specifically, the cameras are 9200b, 9200d, 9200f, 9200h, 9200l, 9200n, 9200r, 9200t, 9200v, 9200x, 9200z, 9200B, 9200D.

The camera selection unit 3016 generates a frame rate change message for the finally selected camera so that the frame rate is set to a high frame rate as in the configuration of the above-described embodiment. The generated frame rate change message is input to each sensor system 110 via the network 270a. The camera adapter 120 of each sensor system 110 receives this and inputs it to the frame rate changing unit 6126 (see FIG. 2), and the frame rate changing unit 6126 changes the frame rate according to the content of the message. The generation of the virtual viewpoint image is also performed with the same configuration and operation as in the first embodiment.

With this configuration, in an image processing system including a plurality of cameras, the cameras that shoot in a predetermined direction are selectively set to a high frame rate. This makes it possible to generate virtual viewpoint content without pixel defects or omissions without changing the transmission band of the normal frame rate. Moreover, at the time of replay, it is possible to browse from the same viewpoint as at the time of live, and further, to browse the virtual viewpoint content from the opposite direction different from that at the time of live.
<Other embodiments>
The camera selection unit described above is configured in the back-end server 270, but is not limited to the configuration. For example, it may be configured in the front-end server 230, and may be configured as long as it can temporarily store and determine the object information and the information of the target background image data as in the first embodiment. Further, the same configuration may be realized by the camera adapter 120, and although the cost varies depending on the number of cameras, it can be processed upstream in the flow of camera image data, so that the camera selection can be determined at a higher speed. ..

Further, as a determination factor for camera selection, in the first embodiment, the execution is performed based on the object of interest and the background image data, but the present invention is not limited to this configuration. For example, the camera may be selected based on the positional relationship between the virtual viewpoint and a plurality of objects. In this case, a virtual camera path for a replay image in which the virtual camera moves with respect to a plurality of objects is generated, and a camera having a high frame rate is selected based on the virtual camera path. Further, a camera may be selected based on the positional relationship of a plurality of objects, and a virtual camera parameter may be generated based on the information to generate a virtual image.

Further, in the above-described embodiment, the interpolated image generation process of the non-existent image data is executed by the virtual viewpoint foreground image generation unit 3005 of the back-end server 270 at the time of the high frame rate. However, the configuration is not limited to this, and the image processing unit 2150 of the front-end server 230 may execute the configuration and output the data to the database 250.

By configuring as described above, as in the above-described embodiment, the camera that shoots the object of interest is selectively set to a high frame rate, so that the transmission band of the normal frame rate is not changed and the virtual image quality is high. It becomes possible to generate viewpoint contents.

Further, in the above-described embodiment, the camera selection unit 3015 selects a camera having a high frame rate. However, the camera selection unit 3015 may select a camera that lowers the frame rate according to the position of the object of interest. In this case, for example, the camera selection unit 3015 selects a camera having no field of view capable of determining the crossing of the ball game as a camera for lowering the frame rate. Further, for example, in FIG. 4, the camera selection unit 3015 may select the camera 9100a and the surrounding cameras farthest on the extended straight line of the field line 9102 as cameras for lowering the frame rate. Further, in this case, the camera selection unit 3015 may send a frame rate change message instructing the camera that lowers the frame rate to lower the frame rate in step S108 of FIG.

Further, in the above-described embodiment, the object of interest has been described as a ball, but the present invention is not limited to this. For example, the object of interest may be a person such as a player or a referee. Further, the object of interest may be a person who exists in a specific area such as an offensive player who exists in the penalty area of the opponent team. In this case, it is possible to shoot a scene where a remarkable event such as a goal or a shot occurs at a high frame rate. Further, the object of interest may be a plurality of objects. Further, the object of interest may be an object that satisfies a specific condition. For example, the player who keeps the ball may be the object of interest.
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

110 sensor system, 111 microphone, 112 camera, 113 cloud stand, 120 camera adapter, 180 switching hub, 190 end user terminal, 230 front end server, 250 database, 270 back end server, 290 time server, 310 control station, 330 virtual Camera operation UI

Claims (15)

Specific means to identify the position of the object of interest taken by multiple shooting devices,
When it is specified by the specific means that the position of the object of interest is included in the specific area, the image transmitted by one or more image pickup devices having a predetermined positional relationship with the specific area among the plurality of image pickup devices. The plurality of imaging devices are controlled so that the frame rate of the data is higher than the frame rate of the image data transmitted by another imaging device different from the one or more imaging devices among the plurality of imaging devices. Control means and
An image processing device characterized by having. The image processing apparatus according to claim 1, wherein the specific area is an area including a specific object different from the object of interest. When the specific means specifies that the object of interest is included in the specific area, the control means sets the frame rate of the image data transmitted by the one or more imaging devices to the other imaging. The image processing apparatus according to claim 1 or 2 , wherein the frame rate is changed so as to be higher than the frame rate of the image data transmitted by the apparatus. The control means transmits information for designating a frame rate of image data transmitted by the one or more photographing devices and the other photographing devices.
The one or more imaging devices and the other imaging devices shoot at a predetermined frame rate, and the image data acquired based on the shooting is transmitted at the frame rate indicated by the information transmitted by the control means. The image processing apparatus according to any one of claims 1 to 3. The control means transmits information for designating a frame rate of image data transmitted by the one or more photographing devices and the other photographing devices.
The image processing according to any one of claims 1 to 4 , wherein the one or more imaging devices and the other imaging devices perform imaging at a frame rate indicated by information transmitted by the control means. Device. The image processing apparatus according to any one of claims 1 to 5, wherein the specific area is a region including a field line in a field photographed by the plurality of photographing devices. The image processing apparatus according to claim 6, wherein the one or more imaging devices are imaging devices installed on an extension of the field line. The specific area according to any one of claims 1 to 7, wherein the specific area includes at least one of a center area, a penalty area, and a goal area in a field photographed by the plurality of photographing devices. Image processing device. The image processing device according to any one of claims 1 to 8 , wherein the object of interest is a moving object photographed by the plurality of photographing devices. The image processing device according to claim 9 , wherein the moving object is a ball or a person photographed by the plurality of photographing devices. The image processing apparatus according to any one of claims 1 to 10 , further comprising a generation means for generating a virtual viewpoint image using a plurality of image data transmitted from the plurality of photographing devices. The image processing apparatus according to claim 11 , wherein the generation means generates a virtual viewpoint image having a frame rate corresponding to the frame rate of the image data transmitted by the one or more photographing devices. The generation means corresponds to the other imaging device at a time when the one or more imaging devices correspond to the frame rate at which the image data is transmitted and the other imaging device does not correspond to the frame rate at which the image data is transmitted. The twelfth aspect of claim 12 , wherein the image data is interpolated based on the image data transmitted from the one or more photographing devices, and a virtual viewpoint image is generated based on a plurality of image data including the interpolated image data. The image processing device described. It is a control method performed by an image processing device.
A specific process to identify the position of the object of interest taken by multiple shooting devices,
When it is specified that the position of the object of interest is included in the specific area in the specific step, the image transmitted by one or more photographing devices having a predetermined positional relationship with the specific area among the plurality of photographing devices. The plurality of imaging devices are controlled so that the frame rate of the data is higher than the frame rate of the image data transmitted by another imaging device different from the one or more imaging devices among the plurality of imaging devices. Control process and
A control method characterized by having. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 13.

Images