JP2019102056A - Image processing device, information processing method and program - Google Patents

Abstract

To provide an image processing device capable of reducing processing load related to a projection of an image to a model.SOLUTION: With respect to an image to be projected on a surface area which is the surface of a background model, which is a model of the background of a virtual viewpoint image generated on the basis of multiple images captured by multiple capture units, the image processing device determines which images of areas in which captured images included in the multiple images captured by multiple imaging units should be projected, and projects the determined images of the areas included in the images captured by multiple imaging units on the surface area.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus, an information processing method, and a program.

Recently, attention has been focused on a technique for generating virtual viewpoint images by arranging imaging devices such as a plurality of cameras at different positions and synchronously capturing images from multiple viewpoints and using images from multiple viewpoints obtained by imaging. ing. A virtual viewpoint image is an image that looks like viewed from a virtually set viewpoint. According to the technology of generating a virtual viewpoint image from images from such a plurality of viewpoints, for example, it becomes possible to view a highlight scene of soccer or basketball from various angles. Generation of a virtual viewpoint image based on images from a plurality of viewpoints can be realized, for example, by performing processing such as foreground / background separation, three-dimensional model generation, and rendering based on images captured by a plurality of cameras.
In Patent Document 1, there is a technology in which a plurality of imaging devices are arranged so as to surround the same range, and a virtual viewpoint image corresponding to any designation is generated and displayed using an image obtained by capturing the same range. It is disclosed.
In the technology for generating a virtual viewpoint image as described above, a background in a virtual viewpoint image can be generated by generating a three-dimensional model such as a stadium having a field serving as a background and projecting the image onto the model. In order to project an image on a model of the background of a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices having different viewpoints, for example, first, projective transformation is performed on the images captured by each imaging device To synthesize. Then, it can be performed by a process of projecting an image synthesized after projective transformation onto a background model.

JP, 2014-215828, A

However, processing relating to projection of an image onto a model (for example, processing for projective transformation of an image captured by an imaging device and processing for combining a captured image of each projection device subjected to projective transformation) requires hardware such as a large amount of memory Because of the need for hardware resources and a large number of operations, techniques for reducing the burden of these processes are desired.
An object of the present invention is to reduce the burden of processing relating to the projection of an image on a model.

  The image processing apparatus according to the present invention projects on a surface area which is an area existing on the surface of a background model which is a model of a background of a virtual viewpoint image generated based on a plurality of photographed images photographed by a plurality of photographing units. Determination means for determining which area of the captured image is to be projected, which is included in the plurality of captured images captured by the plurality of capturing units as the image to be captured; and captured by the plurality of capturing units And projection means for projecting an image of the area determined by the determination means contained in a plurality of photographed images on the surface area.

  According to the present invention, it is possible to further reduce the burden of processing relating to the projection of an image on a model.

FIG. 1 is a diagram illustrating an example of a system configuration of an image processing system. It is a figure which shows an example of the hardware constitutions of an imaging device. It is a figure which shows an example of the hardware constitutions etc. of a server apparatus. It is a flow chart which shows an example of pre-processing. It is a figure explaining an example of a background model. It is a figure explaining an example of the arrangement situation of an imaging device. It is a figure explaining an example of coordinate conversion. It is a flowchart which shows an example of the background production | generation process of a virtual viewpoint image. It is a figure which shows an example of the condition etc. which were image | photographed by the imaging device. It is a figure explaining an example of logging of an image. It is a figure showing an example of functional composition of an imaging device. It is a figure which shows an example of a function structure of a server apparatus. It is a flow chart which shows an example of pre-processing. It is a flow chart which shows an example of background generation processing. It is a flow chart which shows an example of field decision processing. It is a figure explaining an example of the photography range of an imaging device. It is a figure explaining an example of the mesh in which the image to be projected is taken. It is a figure explaining an example of the physical relationship of a standard imaging device and other imaging devices. It is a figure which shows an example of the imaging device put in order according to distance with a reference | standard imaging device. It is a figure explaining an example of the mesh in which the image to be projected is taken. It is a flow chart which shows an example of pre-processing. It is a flow chart which shows an example of background generation processing. It is a figure explaining an example of direction with a virtual viewpoint and each imaging device. It is a flowchart which shows an example of the process at the time of detecting a malfunction.

  Hereinafter, an example of an embodiment of the present invention will be described in detail based on the drawings.

First Embodiment
FIG. 1 is a diagram illustrating an example of a system configuration of an image processing system 100 that generates a virtual viewpoint image. The image processing system 100 is a system that captures an image used to generate a virtual viewpoint image, and generates a virtual viewpoint image from the captured image. The image processing system 100 includes imaging devices 101 a to 101 n and a server device 103. The imaging devices 101 a to 101 n and the server device 103 are communicably connected to each other via the communication line 102.
The imaging devices 101a to 101n are imaging devices having a communication function such as a network camera. Hereinafter, the imaging devices 101 a to 101 n are collectively referred to as an imaging device 101. In the present embodiment, the imaging device 101 is assumed to be 14 imaging devices, but may be 13 or less imaging devices or 15 or more imaging devices. In the present embodiment, the imaging devices 101a to 101n are arranged in a stadium, but may be arranged in other places such as a concert hall. The server apparatus 103 is an information processing apparatus such as a personal computer, a server apparatus, and a tablet apparatus. The server device 103 aggregates the images captured by the imaging devices 101a to 101n, and generates a virtual viewpoint image based on the aggregated images. The communication line 102 is a communication line such as Ethernet. The server device 103 may be configured by a plurality of devices.

FIG. 2A illustrates an example of the hardware configuration of each of the imaging devices 101.
The imaging apparatus 101 includes a CPU 201, a main storage device 202, an auxiliary storage device 203, a network I / F 204, and an imaging unit 205. The elements are communicably connected to each other via a system bus 206.
The CPU 201 is a central processing unit that controls the imaging device 101. The main storage device 202 is a storage device that functions as a work area of the CPU 201 and a temporary storage area of data. The main storage device 202 is implemented using a storage medium such as Random Access Memory (RAM). The auxiliary storage device 203 is a storage device that stores various programs, various setting information, various image data, information of camera parameters, and the like. The auxiliary storage device 203 is mounted using a storage medium such as a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD) or the like.
The network I / F 204 is an interface used for communication with the external apparatus such as the server apparatus 103 via the communication line 102. The imaging unit 205 is an imaging unit that includes an imaging device such as a CMOS sensor and a CCD sensor, a lens, and the like, and images the surroundings.
The CPU 201 executes the processing according to the program stored in the auxiliary storage device 203, whereby the functions of the imaging device 101 described later in FIGS. 2B and 11 and the functions of the imaging device 101 in the flowcharts described later in FIGS. Processing etc. are realized.

FIG. 2B is a diagram illustrating an example of a functional configuration of each of the imaging devices 101.
The imaging apparatus 101 includes an imaging control unit 211, a generation unit 212, and a transmission unit 213.
The imaging control unit 211 controls the imaging unit 205, and converts an optically captured image into digital data by a lens, an imaging element, or the like in the imaging unit 205. The generation unit 212 separates the foreground and the background from the image captured by the imaging control unit 211, and generates a background image excluding the foreground image. The transmission unit 213 transmits the data of the background image generated by the generation unit 212 to the server apparatus 103 via the communication line 102. Note that part or all of the functional configuration shown in FIG. 2B may be realized by hardware. Examples of this hardware include an ASIC, an FPGA, and a GPU.

FIG. 3A is a diagram showing an example of the hardware configuration of the server apparatus 103. As shown in FIG.
The server device 103 includes a CPU 301, a main storage device 302, an auxiliary storage device 303, and a network I / F 304. The elements are communicably connected to each other via a system bus 305.
The CPU 301 is a central processing unit that controls the server device 103. The main storage device 302 is a storage device that functions as a work area of the CPU 301 and a temporary storage area of data. The main storage device 302 is implemented using a storage medium such as a RAM. The auxiliary storage device 303 is a storage device that stores various programs, various setting information, various image data, and the like. The auxiliary storage device 303 is mounted using a storage medium such as a ROM, an HDD, or an SSD. The network I / F 304 is an interface used for communication with the external apparatus such as the imaging apparatus 101 via the communication line 102.
The CPU 301 executes the processing according to the program stored in the auxiliary storage device 303, whereby the functions of the server device 103 described later with reference to FIGS. 3B and 12 and the server device 103 in the flowchart described with reference to FIGS. Processing is realized. In addition, the CPU 301 executes the processing according to the program stored in the auxiliary storage device 303, whereby the processing of the flowchart to be described later is realized in FIGS. 4, 13, 15, 21, 22, and 24.

FIG. 3B is a diagram showing an example of a functional configuration of the server apparatus 103. As shown in FIG.
The server apparatus 103 includes an image acquisition unit 311, a calibration unit 312, a background model management unit 313, a coordinate conversion unit 314, a background texture determination unit 315, a virtual camera control unit 316, and a rendering unit 317.
The image acquisition unit 311 acquires an image captured by each of the imaging devices 101 transmitted from each of the imaging devices 101 via the communication line 102. The image acquisition unit 311 transmits the acquired image to the background texture determination unit 315.
The calibration unit 312 controls each camera parameter of the imaging device 101. The camera parameter is a parameter related to the imaging device, and may include an external parameter that is a parameter related to the position and orientation of the imaging device, and an internal parameter related to the focal length and the pixel interval. Also, the camera parameter may be information for specifying a shooting area. Also, the camera parameters may be only one of the external parameters and the internal parameters, or may include other parameters. The calibration unit 312 transmits, for example, an instruction to change a camera parameter to the imaging apparatus 101 via the communication line 102. The imaging apparatus 101 changes its own camera parameters in accordance with the transmitted instruction. The calibration unit 312 can also receive information on camera parameters from the imaging device 101 via the communication line 102, for example. The calibration unit 312 stores and manages camera parameters of each of the imaging devices 101 in the auxiliary storage device 303, for example. The calibration unit 312 transmits the camera parameters of the imaging apparatus 101 to be managed to the coordinate conversion unit 314.

The background model management unit 313 is data of a background model defined by approximating a three-dimensional mesh model of a structure such as a stadium as a background in a virtual viewpoint image generated by the image processing system 100 (hereinafter, background model Manage data). The data structure of the background model data may be anything, and may indicate a three-dimensional shape by a point, a line segment, a surface or the like. Background model data is stored in advance in the auxiliary storage device 303. The background model management unit 313 transmits the background model data to be managed to the background texture determination unit 315, the coordinate conversion unit 314, and the rendering unit 317. In the present embodiment, it is assumed that the background model management unit 313 manages the coordinates of the background model using a predetermined world coordinate system.
The coordinate conversion unit 314 sets an area in the background model in the world coordinate system to an area in the image captured by the imaging device 101 (an area in the camera coordinate system) based on the camera parameters transmitted from the calibration unit 312. Convert. Further, the coordinate conversion unit 314 converts the area of the camera coordinate system in the image captured by the imaging device 101 into the area in the background model in the world coordinate system based on the camera parameters transmitted from the calibration unit 312. Do. The camera coordinate system is a coordinate system based on the imaging device. Further, the coordinate conversion unit 314 may specify the correspondence between the world coordinate system and the camera coordinate system in each of the imaging devices 101 (for example, conversion parameters between coordinate systems). In that case, the coordinate conversion unit 314 transmits information on the specified correspondence to the background texture determination unit 315. Then, based on the transmitted correspondence information, the background texture determination unit 315 converts the area in the background model in the world coordinate system into the area of the camera coordinate system in the image captured by the imaging device 101. . Also, the background texture determination unit 315 converts the area of the camera coordinate system in the image captured by the imaging device 101 into the area in the background model in the world coordinate system based on the transmitted correspondence information. .

The background texture determination unit 315 determines an image to be projected on each mesh of the background model based on the image acquired by the image acquisition unit 311.
The virtual camera control unit 316 manages information of the virtual camera, and controls camera parameters of the virtual camera based on an operation by the user via the operation unit of the server apparatus 103. The virtual camera is a virtual imaging device considered to be disposed at a designated position. Information of the virtual camera is stored in advance in the auxiliary storage device 303 or the like. The virtual camera control unit 316 transmits information on the virtual camera to the rendering unit 317.
The rendering unit 317 projects the image determined by the background texture determination unit 315 on each mesh of the background model transmitted from the background model management unit 313. Then, the rendering unit 317 renders as a two-dimensional image based on the camera parameters of the virtual camera transmitted from the virtual camera control unit 316. Thus, the rendering unit 317 can generate an image from an arbitrary designated viewpoint based on an instruction of the user via the operation unit of the server apparatus 103.

The determination unit 318 selects an image used to generate a virtual viewpoint image from the images acquired by the image acquisition unit 311. In the auxiliary storage device 303, for example, a specific image such as an out-of-focus image, an image whose exposure is not appropriate (for example, not within a predetermined range), an image captured by a designated imaging device, etc. Information indicating that the virtual viewpoint image is not used is stored in advance. The determination unit 318 selects an image used to generate a virtual viewpoint image from the image acquired by the image acquisition unit 311 based on this information, for example. In the present embodiment, it is assumed that the auxiliary storage device 303 stores information indicating that an image captured by a designated one of the imaging devices 101 is not used for generation of a virtual viewpoint image. In addition, the image processing system 100 may treat an imaging device that captures an image that is determined not to be used for generating a virtual viewpoint image by the determination unit 318 as one that does not exist.
The area determination unit 319 determines an area in which an image to be projected on the background model by the background texture determination unit 315 is captured from the area in the image determined by the determination unit 318 to be used for generating the virtual viewpoint image. Note that part or all of the functional configuration shown in FIG. 3B may be realized by hardware. Examples of this hardware include an ASIC, an FPGA, and a GPU.

FIG. 4 is a flowchart showing an example of pre-processing performed by the image processing system 100 before generation of a virtual viewpoint image. By the process of FIG. 4, the image processing system 100 can reduce the burden of the process of projecting an image onto a background model when generating a virtual viewpoint image.
In step S <b> 401, the background texture determination unit 315 extracts one mesh constituting the background model from the background model data received from the background model management unit 313. Each mesh of the background model is an example of a surface area which is an area present on the surface of the background model. FIG. 5 is a diagram for explaining an example of a background model. The model 501 is a three-dimensional model of a stadium, and a plurality of meshes represent the ground which is a field. In the present embodiment, the background model is a stadium model similar to the model 501. However, the background model may be a model such as a concert hall or a spectator seat. The background model data may include information such as vertex coordinates and the normal of a surface for each of the meshes constituting the ground. In the present embodiment, it is assumed that the background texture determination unit 315 extracts the mesh 502 when the process of S401 is performed for the first time after the process of FIG. 4 is started. In S401, a partial region of the surface of the background model may be extracted in units different from the mesh.

In step S <b> 402, the background texture determination unit 315 selects one from the imaging device 101. In the present embodiment, the background texture determination unit 315 selects the imaging device 101a when performing the process of S402 for the first time after the process of S401.
FIG. 6 is a diagram for explaining an example of the arrangement state of the imaging device. The situation in FIG. 6 shows the positional relationship between a model 501 indicating a stadium to be photographed in the world coordinate system and each of the imaging devices 101 arranged at a position corresponding to the actual arrangement position in the world coordinate system. The imaging device 101a is disposed at the position shown in FIG. 6 and captures an image of the area 601, as indicated by the camera parameters of the imaging device 101a managed by the calibration unit 312. The coordinate conversion unit 314 converts the coordinates of the mesh 502 in the world coordinate system into coordinates in the camera coordinate system of the imaging device 101a.

In step S403, the coordinate conversion unit 314 performs the following process based on the camera parameters of the imaging apparatus 101a transmitted from the calibration unit 312. That is, the coordinate conversion unit 314 converts the area of the mesh 502 extracted in S401 in the world coordinate system into an area in the camera coordinate system of the imaging device 101a. The converted area is an area where a portion corresponding to the mesh 502 in the image captured by the imaging device 101a is captured. That is, the coordinate conversion unit 314 identifies an area in which a portion corresponding to the mesh 502 in each image captured by each of the imaging devices 101 is captured. The situation in which the area of the mesh 502 in the world coordinate system is converted to the camera coordinate system of the imaging device 101a will be described using FIG. An area 701 is an area occupied by the entire image captured by the imaging device 101 a in the camera coordinate system of the imaging device 101 a. FIG. 7 shows a state in which the area of the mesh 502 is converted by the coordinate conversion unit 314 into an area 702 in the camera coordinate system of the imaging device 101a.
In step S404, the coordinate conversion unit 314 determines whether the process of step S403 on the mesh extracted in step S401 is completed for all the imaging apparatuses 101. If the coordinate conversion unit 314 determines that the process of S403 on the mesh extracted in S401 has been completed for all the imaging apparatuses 101, the process proceeds to the process of S405. If the coordinate conversion unit 314 determines that there is an imaging device for which the process of S403 on the mesh extracted in S401 is not completed among the imaging devices 101, the process proceeds to the process of S402.

In step S405, the region determination unit 319 performs the following processing based on the region of the mesh extracted in step S401 converted to the camera coordinate system of each of the imaging devices 101 in step S403. That is, the region determination unit 319 determines a region in which an image projected on a portion of the mesh extracted in S401 in the background model is captured from the regions in the plurality of images captured by each of the imaging devices 101.
The region determination unit 319 is, for example, all of the mesh regions extracted in S401 converted to the camera coordinate system of each of the imaging devices 101 in S403, all contained in the image captured by each of the imaging devices 101 (image Identify the area that does not extend from the Then, the region determination unit 319 selects one from the specified regions, and selects the selected region as a region in which an image projected on the portion of the mesh extracted in S401 in the background model is captured. That is, in the image processing system 100, the image of the selected area in the image captured by each of the imaging devices 101 is projected onto the mesh portion extracted in S401.
In addition, for example, the area determination unit 319 identifies an area that is first identified as an area that is all contained in the image captured by each of the imaging devices 101 among the areas converted in S403, and performs the following processing. You may do it. That is, the area determination unit 319 may select the specified area as an area where the image projected on the part of the mesh extracted in S401 is captured.

In addition, the region determination unit 319 may specify, for example, among the regions converted in S403, regions that are all contained in the image captured by each of the imaging devices 101, and may perform the following processing. That is, the area determination unit 319 selects one from the specified areas based on the area of each area (the number of pixels occupied in the image), and projects the selected area onto the mesh portion extracted in S401. You may select as an area | region where an image is taken.
For example, the region determination unit 319 specifies, among the regions converted in S403, regions that are all contained in the image captured by each of the imaging devices 101. Then, the area determination unit 319 selects one having the largest area from the specified areas, and selects the selected area as an area where the image projected on the portion of the mesh extracted in S401 is captured. Good. By this processing, the image processing system 100 can select the imaging device that captures the portion of the mesh extracted in S401 with the largest resolution as the imaging device that captures the image projected onto the portion. Thus, the image processing system 100 can project a fine image by each mesh of the background model.
In addition, for example, the region determination unit 319 specifies, among the regions converted in S403, regions that are all contained in the image captured by each of the imaging devices 101. Then, the area determination unit 319 selects one having the smallest area from the specified areas, and selects the selected area as an area where the image projected on the mesh portion extracted in S401 is to be photographed. Good. By this processing, the image processing system 100 can minimize the size of the image projected on each mesh of the background model, and can reduce the processing load related to the projection processing. In addition, when the server apparatus 103 receives only a part of the image projected from the imaging apparatus 101 to the mesh of the background model, the communication band of the communication line 102 can be saved. Further, for example, the area determination unit 319 is an area in which an image projected on a portion of the mesh extracted at 401 is taken as an area captured from a plurality of areas among the areas all contained in the image captured by each of the imaging devices 101. It may be specified. In this case, when generating the virtual viewpoint image, the image processing system 100 may project an image obtained by averaging a plurality of images corresponding to a plurality of specified areas onto a corresponding mesh.
In the following, the information of the area selected in S405 is taken as a texture area.

In step S406, the region determination unit 319 determines whether the processing in steps S402 to S405 is complete for all meshes in the background model. If the region determination unit 319 determines that the processes of S402 to S405 have been completed for all meshes in the background model, the process proceeds to the process of S407, and meshes for which the processes of S402 to S405 have not been completed in the background model. If it is determined that there is, the process proceeds to the process of S401.
In step S407, the background texture determination unit 315 transmits, to the rendering unit 317, information (hereinafter referred to as region information) of the texture region selected in step S405 for each mesh in the background model.
The process of FIG. 4 is ready to generate an image to be projected onto the background model.

FIG. 8 is a flowchart illustrating an example of background generation processing of a virtual viewpoint image.
In step S801, the imaging control unit 211 of each of the imaging apparatuses 101 captures an object.
FIG. 9 is a diagram illustrating an example of a situation or the like captured by the imaging apparatus 101. An image 901 is an image captured by the imaging device 101a. In the image 901, persons 902, 902, and 903 are shown as the foreground.
In step S802, the generation unit 212 of each of the imaging devices 101 separates the foreground and the background of the image captured in step S801, and generates a background image from which the foreground image has been removed. In the example of FIG. 9, the generation unit 212 separates persons 902, 902, and 903 that are the foreground from the image 901, and generates a background image 905. The generation unit 212 separates the foreground and the background by using, for example, a method such as a method in which the movement in the time direction has a threshold value or more as the foreground.
In step S <b> 803, the transmission unit 213 of each of the imaging apparatuses 101 transmits the background image generated in step S <b> 802 to the server apparatus 103 via the communication line 102.

In step S804, the background texture determination unit 315 determines an image to be projected on each mesh of the background model of the virtual viewpoint image based on the background image transmitted by each of the imaging apparatuses 101 in step S803. In the present embodiment, the background texture determination unit 315 determines each of the background images transmitted in S803 as an image including an image to be projected on each mesh of the background model, and transmits the image to the rendering unit 317.
In step S805, the rendering unit 317 extracts an image of the area indicated by the area information generated in the process of FIG. 4 from the background image transmitted in step S804, and projects the extracted image on each mesh of the corresponding background model. Do. The rendering unit 317 extracts an area to be projected on each mesh of the background model from the background image, for example, based on the area information. Then, the rendering unit 317 projects the image of the extracted area onto the mesh of the corresponding background model. Thereby, the corresponding image is projected on each mesh of the background model.
Then, the rendering unit 317 generates a virtual viewpoint image by generating an image in which the background model is viewed from the viewpoint of the virtual camera based on the camera parameters of the virtual camera transmitted from the virtual camera control unit 316.

As described above, in the present embodiment, the image processing system 100 determines in advance the regions in which the images to be projected on the meshes of the background model are captured in the images captured by the respective imaging devices 101 in the processing of FIG. . Then, the image processing system 100 generates a virtual viewpoint image by projecting an image captured in the determined region onto each mesh of the background model. Thus, the image processing system 100 projects an image captured in a predetermined area on each mesh of the background model. As a result, the image processing system 100 reduces the process of combining a plurality of images captured by each of the imaging devices 101 after projective transformation. In addition, the image processing system 100 combines a plurality of images captured by each of the imaging devices 101 after projective transformation, and combines the images into a memory such as the main storage device 302 or the CPU 301 as compared to the case where the combined image is projected onto a background model. The burden of processing such as the usage rate of
In addition, the image processing system 100 determines in advance an area in which an image to be projected on each mesh of the background model is captured, and in the image captured by the imaging device 101 that changes with time, the area The image was projected onto each mesh of the background model. As a result, the image processing system 100 can maintain the memory of the main storage device 302 or the like even when the image captured by the imaging device 101, which is the image of the projection target onto each mesh of the background model, changes over time. The burden of processing such as the usage rate of the CPU 301 can be reduced.

Second Embodiment
In the present embodiment, processing of the image processing system 100 in the case where the background texture determination unit 315 cuts out an image to be projected on the mesh of the background model from the background image and transmits the image to the rendering unit 317 will be described.
The system configuration of the image processing system 100 of the present embodiment is the same as that of the first embodiment. Further, the hardware configuration and the functional configuration of each of the imaging devices 101 are the same as in the first embodiment. Furthermore, the hardware configuration and the functional configuration of the server apparatus 103 are the same as in the first embodiment.

Among the processes of the image processing system 100 according to the present embodiment, differences from the first embodiment will be described with reference to FIGS. 4 and 8.
In the present embodiment, in S804, the background texture determination unit 315 cuts out the image of the area indicated by the area information indicating the area determined in the process of FIG. 4 among the background images transmitted by the imaging device 101 in S803. . Then, the background texture determination unit 315 determines the extracted image as an image to be projected on each mesh of the background model of the virtual viewpoint image. The background texture determination unit 315 transmits the determined image to the rendering unit 317.

FIG. 10 is a diagram for explaining an example of clipping of an image by the background texture determination unit 315. An area 1000 is a rectangular area including an area 702 which is an area of the mesh 502 (in the camera coordinate system of the imaging device 101a) in the image captured by the imaging device 101a, which is obtained in the processing up to S405. The imaging apparatus 101 a is assumed to be an imaging apparatus determined by the determination unit 318 as an imaging apparatus for capturing an image projected on the mesh 502. The background texture determination unit 315 identifies a rectangular area inscribed from the area 701 to the area 702, and determines the identified area 1001 as an area to be extracted. In the example of FIG. 10, the background texture determination unit 315 cuts out the range of the area 1001 in the background image generated from the image captured by the imaging device 101a in S804, and acquires the cut out image 1002.
In step S815, the rendering unit 317 projects the image of the region where the image projected onto the mesh included in the image transmitted in step S804 is captured onto each mesh of the corresponding background model. Then, the rendering unit 317 performs rendering based on the camera parameters of the virtual camera specified by the virtual camera control unit 316, and generates a virtual viewpoint image.

  As described above, in the present embodiment, the background texture determination unit 315 cuts out the image to be projected on the mesh from the background image, and transmits the cut out image to the rendering unit 317. Thereby, the image processing system 100 can reduce the size of data to be transmitted from the background texture determination unit 315 to the rendering unit 317 as compared to the case where the entire background image is transmitted.

Embodiment 3
In the present embodiment, an image processing system in which each of the imaging devices 101 cuts out an image of a region corresponding to each mesh of the background model from the background image of the image captured by each of the imaging devices 101 and transmits it to the server device 103 The process of 100 will be described.
The system configuration of the image processing system 100 of the present embodiment is the same as that of the first embodiment. Further, the hardware configuration of each of the imaging devices 101 is the same as that of the first embodiment. The hardware configuration of the server apparatus 103 is the same as that of the first embodiment.

FIG. 11 is a diagram showing an example of the functional configuration of each of the imaging devices 101 of the present embodiment.
The imaging apparatus 101 according to the present embodiment differs from that in FIG. 2B in that it includes a cutout area reception unit 1101 and a cutout unit 1102.
The cutout area reception unit 1101 receives, from the server apparatus 103, information on areas corresponding to the meshes of the background model in the image captured by the imaging apparatus 101. In the present embodiment, in step S407, the background texture determination unit 315 transmits the area information to the imaging apparatus 101 that captures an area indicated by the area information. The cutout area receiving unit 1101 receives the transmitted area information.
The cutout unit 1102 cuts out an image from the background image generated by the generation unit 212 based on the area information received by the cutout area reception unit 1101, and transmits the cutout image to the server apparatus 103 via the transmission unit 213. Send to

FIG. 12 is a diagram showing an example of a functional configuration of the server apparatus 103 according to the present embodiment.
The server apparatus 103 according to the present embodiment includes an image acquisition unit 311, a calibration unit 312, a background model management unit 313, a coordinate conversion unit 314, a background texture determination unit 315, a virtual camera control unit 316, a rendering unit 317, a determination unit 318, A region determination unit 319 is included. Further, the server device 103 according to the present embodiment includes a cutout area transmission unit 1201.
The calibration unit 312, the background model management unit 313, the coordinate conversion unit 314, the virtual camera control unit 316, the determination unit 318, and the area determination unit 319 are the same as those in FIG. 3B.
The background texture determination unit 315 has a function of transmitting area information to each of the imaging devices 101 via the cutout area transmission unit 1201, in addition to the same function as that of the first embodiment.
The cutout area transmission unit 1201 transmits the area information received from the background texture determination unit 315 to each of the imaging devices 101 via the communication line 102.
The image acquisition unit 311 receives an image cut out from the background image received from each of the imaging devices 101 via the communication line 102, and transmits the image to the rendering unit 317.
The rendering unit 317 projects the extracted image received by the image acquisition unit 311 on each mesh of the background model using the area information received from the background texture determination unit 315 to generate a virtual viewpoint image.

FIG. 13 is a flowchart showing an example of pre-processing performed by the image processing system 100 according to the present embodiment before generating a virtual viewpoint image.
The processing of S401 to S405 in FIG. 13 is the same as that of FIG. Among the processing of FIG. 13, points different from FIG. 4 will be described.
In step S1301, the cutout area transmission unit 1201 receives the area information generated by the background texture determination unit 315, and transmits the received area information to the corresponding imaging apparatus 101 via the communication line 102. As in the example of FIG. 7, when the area 702 is selected by the area determination unit 319 in S405, the background texture determination unit 315 determines the area 1001 of FIG. 10 as the area to be extracted and indicates the determined area. The area information is transmitted to the cutout area transmission unit 1201. Then, the cutout area transmission unit 1201 transmits the area information indicating the area 1001 to the imaging device 101a that captures the area 1001.

FIG. 14 is a flowchart illustrating an example of background generation processing of a virtual viewpoint image performed by the image processing system 100 according to the present embodiment. The processes of S801 and S802 in FIG. 14 are the same as those in FIG. Among the processing of FIG. 14, points different from FIG. 8 will be described.
In S1401, the cutout unit 1102 cuts out the area indicated by the area information received by the cutout area receiving unit 1101 from the background image generated in S802. When the area information indicates the area 1001, the cutout unit 1102 cuts out the range of the area 1001 from the background image, and generates the image 1002.
In S1402, the transmission unit 213 transmits the image cut out by the background image cutting unit 1102 in S1401 to the server apparatus 103.
In S1403, the rendering unit 317 projects the image transmitted in S1402 on the meshes of the corresponding background model. Then, the rendering unit 317 performs rendering based on the camera parameters of the virtual camera specified by the virtual camera control unit 316, and generates a virtual viewpoint image.

  As described above, in the present embodiment, each of the imaging devices 101 executes the processing of cutting out the image projected on the mesh of the background model from the background image performed by the server device 103 in the second embodiment. Thus, the image processing system 100 can reduce the processing load of the server apparatus 103. Further, the imaging apparatus 101 transmits not the entire background image but the clipped image to the server apparatus 103. Therefore, the image processing system 100 can save the communication bandwidth of the communication line 102.

Fourth Embodiment
In the present embodiment, a process in which the image processing system 100 determines an area based on camera parameters of each of the imaging apparatuses 101 in S405 will be described.
The system configuration of the image processing system 100 of the present embodiment is the same as that of the first embodiment. Further, the hardware configuration and the functional configuration of each of the imaging devices 101 are the same as in the first embodiment. Furthermore, the hardware configuration and the functional configuration of the server apparatus 103 are the same as in the first embodiment.
The process of the present embodiment is different from that of the first embodiment in that the image processing system 100 performs the process of FIG. 15 instead of the processes of S401 to S406 of FIG. Points different from the first embodiment will be described.

FIG. 15 is a flowchart showing an example of the area determining process by the area determining unit 319 according to the present embodiment.
In step S <b> 1501, the region determination unit 319 determines, from the imaging device 101, an imaging device serving as a reference of determination of a region where an image projected onto the mesh of the background model is captured. Hereinafter, an imaging device serving as a reference for determining a region in which an image projected onto a mesh of a background model is captured will be referred to as a reference imaging device. The reference imaging device is an example of a reference imaging unit.
FIG. 16 is a diagram for explaining an example of the imaging range of the imaging device. An area 601 indicates the imaging range for the background model of the imaging device 101a, as in FIG. An area 1601 indicates the imaging range for the background model of the imaging device 101 e having the widest field of view in the imaging device 101. In the present embodiment, the region determination unit 319 determines the imaging device 101e having the widest field of view in the imaging device 101 as a reference imaging device in S1501. However, the area determination unit 319 may determine an imaging apparatus capable of capturing an area corresponding to the mesh of the largest number of background models in the imaging apparatus 101 as the reference imaging apparatus in S1501. In addition, the region determination unit 319 may determine an imaging device predetermined in the imaging device 101 as the reference imaging device in S1501. In addition, the region determination unit 319 may determine the designated imaging device as the reference imaging device based on the user's operation via the operation unit of the server device 103 in S1501.

In step S1502, the region determination unit 319 specifies, among the meshes of the background model, meshes that are to be included in the image captured by the reference imaging device as the entire mesh converted into the camera coordinate system of the reference imaging device. Then, the area determination unit 319 projects the areas corresponding to the identified meshes (areas in which the meshes are converted to the camera coordinate system) in the image captured by the reference imaging device determined in S1501 onto the meshes. Is determined as the area in which the image is captured.
FIG. 17 is a view for explaining an example of a mesh on which an image projected by the reference imaging device is captured. A hatched area 1701 is a mesh area on which an image projected by the reference imaging device is captured. The meshes in the other areas are meshes in which the area in which the projected image is to be captured has not been determined.
In step S <b> 1503, the region determination unit 319 determines, for all meshes in the background model, whether a region in which an image to be projected is captured has been determined. If the area determination unit 319 determines that the area in which the projected image is captured is determined for all meshes in the background model, the process of FIG. 15 ends, and the process proceeds to step S407. If the area determination unit 319 determines that there is a mesh in which the area in which the projected image is to be captured is not determined in the meshes in the background model, the process proceeds to the process of S1504.
When the reference imaging apparatus is the imaging apparatus 101e and the process of S1503 is performed for the first time after the process of S1502, the area determining unit 319 determines the area where the image to be projected is still captured as shown in FIG. Since there is no mesh, the process proceeds to the process of S1503.

In step S1504, the area determination unit 319 captures an area in which an image to be projected on the mesh of the background model is captured based on camera parameters of the reference imaging apparatus and another imaging apparatus in the imaging apparatus 101. Select a candidate. In the following, the imaging device selected in the process of the most recent S1504 is referred to as a selected imaging device.
FIG. 18 is a diagram for explaining an example of the positional relationship between a reference imaging device and another imaging device. The area determination unit 319 acquires the distance between the coordinates (one of the camera parameters) in the world coordinate system of the imaging device 101e which is the reference imaging device and the coordinates in the world coordinate system of each of the other imaging devices 101. In the example of FIG. 18, the imaging device 101 e which is a reference imaging device and the imaging device 101 f are separated by a distance 1801. Further, the imaging device 101e, which is a reference imaging device, and the imaging device 101g are separated by a distance 1802. Further, the imaging device 101e, which is a reference imaging device, and the imaging device 101h are separated by a distance 1803.
FIG. 19 is an example of a table in which each of the imaging devices 101 excluding the reference imaging device is arranged in order of increasing distance from the reference imaging device. In the example of FIG. 19, the image pickup apparatus 101 f is closest to the reference image pickup apparatus. Therefore, when the process of S1504 is performed for the first time after the process of FIG. 15 is started, the area determination unit 319 starts an imaging apparatus 101f having a value closest to the reference imaging apparatus with respect to the coordinates in the world coordinate system that is camera parameters select. Also, when performing the processing of S1504 that follows, the region determination unit 319 is the closest to the reference imaging device with regard to the coordinates in the world coordinate system, which are camera parameters, from among the imaging devices 101 that have not been selected in S1504. An imaging device having a value will be selected.

In step S1505, the region determination unit 319 is a mesh that can capture the entire region corresponding to the mesh by the selective imaging device among the meshes of the background model in which the region on which the projected image is not captured is not determined. Identify Then, the area determination unit 319 determines an area in which the specified mesh area is converted to the camera coordinate system of the selected imaging device as an area in which an image projected on the mesh is captured. Then, the region determination unit 319 proceeds to the process of S1503. The background texture determination unit 315 transmits the information on the area determined by the area determination unit 319 as the area information to the rendering unit 317 by the process of FIG.
FIG. 20 is a view for explaining an example of a mesh in which an image to be projected is photographed by the selective imaging device. In the example of FIG. 20, the selected imaging device is the imaging device 101f. An area 2001 indicates the imaging range of the imaging device 101 f. Among meshes included in the area 2001, meshes not included in the area 1701 are meshes on which an image to be projected is captured by the imaging device f. In the example of FIG. 20, the meshes included in each of the areas 2002 to 2004 are meshes on which the projected image is captured by the imaging device f.

As described above, in the present embodiment, the image processing system 100 determines the reference imaging device, and specifies the mesh in which the entire corresponding region is photographed by the reference imaging device among the meshes of the background model. Then, the image processing system 100 determines, for the identified meshes, the regions corresponding to those meshes in the image captured by the reference imaging device as the regions in which the images projected onto those meshes are captured.
Then, the image processing system 100 selects one imaging device from the imaging device 101 as the selected imaging device in order from the camera parameter closest to the reference imaging device as the selected imaging device and performs the following processing. That is, the image processing system 100 determines an area in which an image to be projected on the mesh of the background model is captured from among the images captured by the selective imaging device. More specifically, the image processing system 100 specifies, among meshes of the background model, an area in which a projected image is photographed is not determined, and a mesh in which the entire corresponding area is photographed by the selective imaging device is specified. Then, the image processing system 100 determines, for the identified meshes, the regions corresponding to those meshes in the image captured by the selective imaging device as the regions in which the images projected onto those meshes are captured. The image processing system 100 has performed the above-described processing for all meshes in the background model until the area where the projected image is captured is determined.
As a result, in the image processing system 100, an image projected on the mesh of the background model is captured by the reference imaging device and the imaging device whose camera parameter is closer to that of the reference imaging device. Therefore, the image processing system 100 can generate a more natural background model in which an image closer to imaging conditions is finally projected to each mesh.

In addition, for all meshes that can be photographed by one imaging device in S1502 and S1505, the image processing system 100 selects, from among the images photographed by the imaging device, regions where the images projected on all the meshes are photographed. Were determined. As a result, an image included in an image captured by the same imaging device is projected onto more adjacent meshes. Thereby, the image processing system 100 can generate a more natural background model in which meshes are more naturally continuous.
In the present embodiment, the image processing system 100 determines the area where the image projected on the mesh of the background model is captured based on the coordinates in the world coordinate system of the reference imaging device and the other imaging devices 101. However, the image processing system 100 determines the area where the image projected on the mesh of the background model is captured based on camera parameters other than the coordinates in the world coordinate system of the reference imaging device and the other imaging devices 101. It is also good. For example, the region determination unit 319 may select the selected imaging device in order from the one closer to the reference imaging device as the exposure value which is the camera parameter in S1504. In addition, in S1504, the region determination unit 319 may select the selected imaging device in order from the one closer to the reference imaging device as the imaging direction which is the camera parameter.

Fifth Embodiment
In the present embodiment, a process will be described in which the image processing system 100 determines, for each mesh of the background model, an area in which an image to be projected is captured, based on camera parameters of each of the virtual camera and the imaging device 101.
The system configuration of the image processing system 100 of the present embodiment is the same as that of the first embodiment. Further, the hardware configuration of each of the imaging devices 101 is the same as that of the first embodiment. The hardware configuration of the server apparatus 103 is the same as that of the first embodiment.
The functional configuration of the server apparatus 103 according to the present embodiment is as shown in FIG. The image acquisition unit 311, the calibration unit 312, the background model management unit 313, and the rendering unit 317 according to the present embodiment are the same as in the first embodiment.

In addition to the functions described in the first embodiment, the virtual camera control unit 316 according to the present embodiment has a function of transmitting camera parameters of the virtual camera to the background texture determination unit 315.
The calibration unit 312 has a function of transmitting camera parameters of each of the imaging devices 101 to the background texture determination unit 315, in addition to the functions described in the first embodiment.
The background texture determination unit 315 uses the camera parameters acquired from the calibration unit 312 for each mesh of the background model acquired from the background model management unit 313 to obtain the area on the image acquired from the image acquisition unit 311 (camera coordinates Convert to the upper area). Then, the background texture determination unit 315 determines an area in which an image projected on the corresponding mesh is to be captured from among the converted areas. At this time, the background texture determination unit 315 has a function of determining a region most suitable for the virtual viewpoint.

FIG. 21 is a flowchart showing an example of pre-processing performed before the server device 103 according to the present embodiment generates a virtual viewpoint image.
In step S <b> 2101, the background texture determination unit 315 selects one of the imaging devices 101.
In step S2102, the region determination unit 319 performs the processing of steps S1502 to S1505 on the assumption that the imaging device selected in step S2101 is a reference imaging device, thereby capturing an image in which a projected image is captured for each mesh of the background model. Decide. Then, the background texture determination unit 315 stores the information of the determined area in the auxiliary storage device 303 or the like as area information corresponding to the imaging device selected in S2101.
In step S <b> 2103, the region determination unit 319 determines whether the processing in step S <b> 2102 is completed for all of the imaging apparatuses 101. If the area determination unit 319 determines that the process of S2102 is completed for all of the imaging apparatuses 101, the area determination unit 319 ends the process of FIG. If the region determination unit 319 determines that some of the imaging apparatuses 101 have not completed the process of S2102, the process proceeds to the process of S2101.
Corresponding area information can be prepared for each of the imaging devices 101 by the process of FIG.

FIG. 22 is a flowchart showing an example of a process of generating a virtual viewpoint image performed by the image processing system according to the present embodiment. In the present embodiment, a reference imaging device that is a reference for determining a region in which an image projected onto a mesh of a background model is captured is a virtual camera.
In step S <b> 2201, the virtual camera control unit 316 receives an input of camera parameters of the virtual camera based on the user's operation via the operation unit of the server apparatus 103.
In step S <b> 2202, the background texture determination unit 315 performs the following process based on the camera parameters received in step S <b> 2201 and the camera parameters acquired from the calibration unit 312. That is, the background texture determination unit 315 obtains an inner product of the vector of the imaging direction which is the camera parameter accepted in S2201 and the vector of the imaging direction which is the camera parameter acquired from the calibration unit 312. FIG. 23 is a diagram for explaining an example of the directions of the virtual camera and the imaging apparatus 101. The virtual camera 2301 is a virtual camera facing in the direction indicated by the vector 2302. Vector 2302 is a vector normalized to have a length of one. Vectors 2303 a to 2303 n are vectors normalized to have a length of 1 indicating the orientation of each of the imaging devices 101. The background texture determination unit 315 obtains an inner product of the vector 2302 and each of the vectors 2303a to 2303n. Each of the vector 2302 and the vectors 2303 a to 2303 n is an example of information indicating a shooting direction which is a camera parameter.

In step S2203, the background texture determination unit 315 specifies the imaging device 101 corresponding to the largest value among the inner products obtained in step S2202. In the example of FIG. 23, the background texture determination unit 315 specifies an imaging device 101k that is facing the vector 2303k most similar to the vector 2302 among the vectors 2303a to 2303n.
In step S2204, the background texture determination unit 315 transmits, to the rendering unit 317, area information corresponding to the imaging device specified in step S2203 and stored in the auxiliary storage device 303 or the like in step S2102.
In step S2205, the rendering unit 317 projects the image indicated by the area information transmitted in step S2204 in the image captured by each of the imaging devices 101 onto each mesh of the corresponding background model. Then, the rendering unit 317 performs rendering based on the camera parameters of the virtual camera specified by the virtual camera control unit 316, and generates a virtual viewpoint image.
In the present embodiment, the image processing system specifies the imaging device using the inner product of the vectors of the directions of the virtual camera and the imaging device 101 in S2203. However, the image processing system may specify the imaging device based on the coordinates of each of the virtual camera and the imaging device 101 in S2202 (for example, based on the distance between each of the virtual camera and each of the imaging device 101). For example, the image processing system may specify an imaging device closest to the virtual camera and use area information corresponding to the specified imaging device instead of the processing of S2202 to S2203. In that case, in the example of FIG. 23, the image processing system specifies, for example, the imaging device 101j closest to the virtual camera 2301.

As described above, in the present embodiment, the image processing system identifies the imaging device that is the closest to the virtual camera as the imaging direction among the imaging devices 101, and the background according to the region information corresponding to the identified imaging device. We projected the image to each mesh of the model.
As a result, the image processing system can use area information according to an imaging device having similar features by the virtual camera, and can make the virtual viewpoint image captured by the virtual camera a more appropriate image.

Embodiment 6
In the present embodiment, a process for coping with a case where a defect that prevents acquisition of an image suitable for generating a virtual viewpoint image from the imaging device 101 included in the image processing system 100 occurs will be described.
The system configuration of the image processing system 100 of the present embodiment is the same as that of the first embodiment. Further, the hardware configuration and the functional configuration of each of the imaging devices 101 are the same as in the first embodiment. Furthermore, the hardware configuration and the functional configuration of the server apparatus 103 are the same as in the first embodiment.
In the following, among the processing of the present embodiment, differences from the first to fifth embodiments will be described.

FIG. 24 is a flowchart illustrating an example of processing when the image processing system 100 detects a defect that prevents acquisition of an image suitable for generating a virtual viewpoint image from the imaging device 101. The process of the present embodiment will be described with reference to FIG. In the following, the problem that it becomes impossible to acquire an image suitable for generating a virtual viewpoint image from the imaging device 101 is simply referred to as a defect. The defects include, for example, a failure of the imaging device 101, a change in position / posture due to strong wind, a change in position / posture due to a collision with a bird or flying object, etc., a failure (for example, disconnection) of the communication line 102, the communication line 102 There is a tight communication band. Further, the problems include occurrence of defocusing in the imaging device 101, unexpected fluctuation of the exposure value in the imaging device 101, and the like.
The image processing system 100 can execute the process of FIG. 24 at any timing. The image processing system 100 may execute the processing of FIG. 24 periodically at predetermined intervals, for example.
In addition, the image processing system 100 may execute the process of FIG. 24 in response to satisfaction of a predetermined condition. The image processing system 100 may execute the process of FIG. 24 only once in response to the predetermined condition being satisfied, and repeats the process of FIG. 24 each time the condition is satisfied. It is good also as execution.
For example, the image processing system 100 may execute the process of FIG. 24 in response to generation of a predetermined number (for example, one or ten) of virtual viewpoint images. In addition, the image processing system 100 may execute the process of FIG. 24 each time a background image is captured by the imaging device 101.
In addition, the image processing system 100 executes the process of FIG. 24 when the determination unit 318 determines that a part or all of the background image captured by each of the imaging devices 101 is not suitable for generating a virtual viewpoint image. You may do it. In the image processing system 100, the process in FIG. 24 is performed in response to the detection unit 318 detecting that a communication trouble has occurred in communication with the imaging device 101 via the network I / F 304 and the communication line 102. It is good also as execution.
In addition, the image processing system 100 may execute the process of FIG. 24 in response to the information indicating the occurrence of a failure being input from the user via the operation unit of the server apparatus 103 or the like.

In step S <b> 2401, the determination unit 318 detects a defect that an image suitable for generating a virtual viewpoint image can not be acquired from the imaging device 101.
For example, the determination unit 318 determines that a failure (for example, a failure, a change in position / posture, occurrence of out-of-focus, or a change in exposure value) has occurred in the imaging device 101 via the network I / F 304, for example. By receiving the indicated information, a fault is detected. In this case, the imaging apparatus 101 detects, for example, a failure (for example, a failure, defocus, fluctuation in exposure value, fluctuation in position / posture, etc.) caused by itself, and transmits the detected failure information to the server apparatus 103. .
Alternatively, the determination unit 318 may detect a failure by detecting that the communication with the imaging apparatus 101 is interrupted for a period equal to or greater than a predetermined threshold. For example, when a packet indicating that communication has been established is not received from the imaging device 101 for a period equal to or greater than a predetermined threshold value, the determination unit 318 detects that communication with the imaging device 101 is interrupted. , And may detect a failure.
In addition, the determination unit 318 detects a defect when information indicating that a defect such as a failure of the imaging device 101 has occurred is input based on an operation by the user via the operation unit of the server device 103 or the like. It may be
In the present embodiment, the main storage device 302 stores failure information indicating a failure that is currently occurring and an imaging device that can not acquire an appropriate background image due to the failure. When the failure does not occur, the determination unit 318 may store, as the failure information, information indicating that the failure has not occurred, or may store empty information.

In S2402, the determination unit 318 compares the defect detected in S2401 with the defect indicated by the defect information stored in the auxiliary storage device 303, and determines whether there is a change in the occurrence status of the defect. If the determination unit 318 determines that there is a change in the occurrence status of the occurring failure, the process proceeds to the processing of S 2403, and if it determines that the occurrence status of the occurring failure does not change, the processing proceeds to the process of S 2405 .
In S 2403, the determination unit 318 updates the defect information stored in the main storage unit 302 so as to indicate the defect detected in S 2401 and the imaging device 101 whose appropriate background image can not be obtained due to the defect. .

The process of S2404 is the process of FIG. 4 which is performed without using the imaging device 101 which can not acquire an appropriate image indicated by the defect information stored in the main storage device 302. The processing of S2404 is different from the processing of FIG. 4 of the first embodiment and in the processing of S402 and S404. S402 and S404 in the process of S2404 will be described. In step S <b> 402, the background texture determination unit 315 executes processing for selecting one of the imaging devices 101 excluding the imaging device 101 that can not acquire an appropriate image indicated by the defect information. Further, in S2404, in S404, the coordinate conversion unit 314 performs the following processing on all the imaging devices 101 except the imaging device 101 that can not acquire an appropriate image indicated by the defect information. That is, the coordinate conversion unit 314 performs a process of determining whether the process of S403 on the mesh extracted in S401 is completed. If the coordinate conversion unit 314 determines that the process has been completed, the process proceeds to the process of S405. If it is determined that the process is not completed, the process proceeds to the process of S402. In this manner, the server apparatus 103 does not use an image captured by the imaging apparatus 101 in a state in which an appropriate image can not be acquired, as an image projected on the mesh of the background model of the virtual viewpoint image.
When the server apparatus 103 executes the process of FIG. 24 for the first time, the background image may be generated by executing the process of S2404 regardless of the result of the process of determination in S2402.

Further, the process of S2404 may be any of the processes of FIGS. 13, 15 and 21 performed without using the imaging apparatus 101 that can not acquire an appropriate image indicated by the defect information stored in the main storage device 302. .
First, when the process of S2404 is the process of FIG. 13 performed without using the imaging apparatus 101 that can not acquire an appropriate image indicated by the defect information, the process of S2404 corresponds to the process of FIG. 13 of the third embodiment and S402. The contents of the processing of step S404 and step S404 are different. The processes of S402 and S404 in the process of S2404 in this case will be described. In step S <b> 402, the background texture determination unit 315 executes processing for selecting one of the imaging devices 101 excluding the imaging device 101 that can not acquire an appropriate image indicated by the defect information. Also, in S404, the coordinate conversion unit 314 completes the process of S403 on the mesh extracted in S401 for all the imaging devices 101 except the imaging device 101 that can not acquire an appropriate image indicated by the defect information. Execute processing to determine whether or not. If the coordinate conversion unit 314 determines that the process has been completed, the process proceeds to the process of S405. If it is determined that the process is not completed, the process proceeds to the process of S402.

Next, when the process of S2404 is the process of FIG. 15 performed without using the imaging apparatus 101 that can not acquire an appropriate image indicated by the defect information, the process of S2404 is the process of FIG. 15 of the fourth embodiment; The contents of the processing of S1501 and S1504 are different. The processes of S1501 and S1504 in the process of S2404 in this case will be described. In step S <b> 1501, the region determination unit 319 executes a process of selecting a reference imaging device from among the imaging devices 101 excluding the imaging device 101 that can not acquire an appropriate image indicated by the defect information. In step S1504, the region determination unit 319 executes a process of selecting an imaging device to be the selected imaging device from among the imaging devices 101 excluding the imaging device 101 that can not acquire an appropriate image indicated by the defect information.
In S1501 in the process of S2404, the region determination unit 319 may select the reference imaging device without excluding the imaging device 101 that can not acquire an appropriate image indicated by the defect information from the candidates. That is, in step S1501, the imaging apparatus 101 in which a problem occurs may be selected as the reference imaging apparatus. In this case, in S1504, based on the positional relationship with the failed reference imaging device, an imaging device to be the selected imaging device is selected from the imaging devices 101 other than the reference imaging device. According to such a selection method of the imaging device 101, the change in the selected imaging device is small before and after the occurrence of the defect as compared with the case where the reference imaging device is changed according to the occurrence of the defect. It is possible to suppress the change from giving a sense of incongruity to the user.
Next, when the process of S2404 is the process of FIG. 21 performed without using the imaging apparatus 101 that can not acquire an appropriate image indicated by the defect information, the process of S2404 is the process of FIG. 21 of the fifth embodiment; The contents of processing of S2101 and S2103 are different. The processes of S2101 and S2103 in the process of S2404 in this case will be described. In step S <b> 2101, the background texture determination unit 315 executes processing for selecting an imaging device from among the imaging devices 101 excluding the imaging device 101 that can not acquire an appropriate image indicated by the defect information. In step S2103, the region determination unit 319 determines whether the processing in step S2102 is complete for all components of the imaging apparatus 101 except the imaging apparatus 101 that can not acquire an appropriate image indicated by the defect information. If the area determination unit 319 determines that the process has been completed, the process of FIG. 21 ends. If the area determination unit 319 determines that the process is not completed, the process proceeds to step S2101.
The process of S2405 is the process of FIG. 8 performed based on the area information (information of an area (texture area) in which an image projected on each mesh of the background model is captured) obtained in the process of S2404. If it is determined in S2402 that there is no change in the occurrence status of the failure, the process of S2405 is performed based on the area information obtained in the process of S2404 in the process of FIG. 24 performed in the past. Also, the process of S2405 may be the process of FIG. 14 performed based on the area information obtained in the process of S2404. Also, the process of S2405 may be the process of FIG. 22 performed based on the area information obtained in the process of S2404.

As described above, in the present embodiment, the image processing system 100 detects a failure that an appropriate image can not be acquired from the imaging device 101. Then, the image processing system 100 excludes the area of the image captured by the imaging device 101 which can not acquire an appropriate image, and determines the area in which the image projected on each mesh of the background model is captured. As a result, the image processing system 100 projects an image captured by the imaging device 101 capable of acquiring an appropriate image on each mesh of the background model even when a failure occurs, thereby making the virtual viewpoint image appropriate Background image can be generated.
Further, in the present embodiment, it is assumed that the range of the background texture that can be generated is narrowed by the imaging device 101 in which the failure has occurred. For example, with regard to the mesh 502 in the first embodiment, FIG. 6 shows that the image is captured by the imaging device 101a. In this case, it is assumed that the mesh 502 is a mesh corresponding to a region which is not photographed in any of the imaging devices 101 other than the imaging device 101a. In that case, there is no imaging device 101 capable of capturing an image projected onto the mesh 502.
In such a case, the image processing system 100 may be as follows. The image acquisition unit 311 stores at least one or more images from the most recent image captured when each of the imaging devices 101 is operating normally, in the main storage device 302, the auxiliary storage device 303, and the like. For example, it is assumed that an area in which imaging can not be performed due to a defect such as a failure in the imaging apparatus 101a and that imaging is not performed by all the other imaging apparatuses 101 is generated. In that case, the image processing system 100 may use, as an image of the area, an image captured in the past by the imaging device 101a stored in the main storage device 302, the auxiliary storage device 303, and the like.

In addition, when there are a plurality of imaging devices 101 that capture an area corresponding to a certain mesh, the image processing system may be as follows. That is, the image acquisition unit 311 stores at least one or more images from the newest image captured when each of the plurality of imaging devices 101 is operating normally in the main storage device 302, the auxiliary storage device 303, etc. You may do it. In this case, even if a failure occurs in part of the plurality of imaging devices 101, the image processing system 100 can acquire an image of the area captured by the remaining imaging devices 101. In addition, even if a problem occurs in all of the plurality of imaging devices 101, the image processing system 100 may use the imaging device 101a stored in the main storage device 302, the auxiliary storage device 303, or the like as an image of the area in the past. The photographed image can be used.
As described above, even when problems occur in all of the imaging devices 101 capturing a certain mesh, the image processing system 100 can generate a background image of a virtual viewpoint image using an image captured in the past.
By such processing, the image processing system 100 can suppress the generation of the background image even when an area which is not seen from anywhere is generated.
Furthermore, before generating the virtual viewpoint image, the image processing system 100 performs imaging in advance by each of the imaging devices 101, and stores the imaged image in the main storage device 302, the auxiliary storage device 303, and the like in advance. It is also good.

<Other Embodiments>
In the first to sixth embodiments, the server apparatus 103 is a single information processing apparatus. However, the server apparatus 103 may be configured by a plurality of information processing apparatuses. In that case, the CPUs of the information processing apparatuses constituting the server apparatus 103 execute processing in cooperation with each other in accordance with the program stored in the auxiliary storage device or the like of the information processing apparatuses constituting the server apparatus 103. Functions and processing are realized. That is, the functions of FIGS. 3B and 12, the processing of the server apparatus 103 in the flowcharts of FIGS. 8 and 14, and the processing of the flowcharts of FIGS. 4, 13, 15, 21, 22, 24 are realized.
The image captured by the imaging device 101 in the first to sixth embodiments is an example of a captured image.
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.
For example, part or all of the functional configuration of the image processing system 100 described above may be implemented as hardware in the imaging apparatus 101 and the server apparatus 103.
As mentioned above, although the example of embodiment of this invention was explained in full detail, this invention is not limited to the specific embodiment which concerns. The above-described embodiments may be arbitrarily combined, appropriately modified or applied.

100 image processing system 101 imaging device 103 server device

Claims (14)

As the image projected on the surface area which is an area existing on the surface of the background model which is a model of the background of the virtual viewpoint image generated based on the plurality of photographed images photographed by the plurality of photographing units, the plurality of photographings A determination unit configured to determine which area of the photographed image included in the plurality of photographed images photographed by the unit is to be projected;
Projection means for projecting an image of the area determined by the determination means included in the plurality of photographed images photographed by the plurality of photographing units onto the surface area;
An image processing apparatus having:   The determination unit is configured to set the surface region to a plurality of regions corresponding to the surface region among the regions in the plurality of photographed images photographed by the plurality of photographing units, based on the respective areas of the plurality of regions. The image processing apparatus according to claim 1, wherein the area of the image to be projected is determined.   The image processing apparatus according to claim 2, wherein the determination unit determines an area having the largest area from the plurality of areas as an area of an image projected onto the surface area.   The image processing apparatus according to claim 2, wherein the determination unit determines an area having the smallest area from the plurality of areas as an area of an image projected onto the surface area.   The determination unit is configured to determine a predetermined reference imaging unit and a plurality of imaging units from a plurality of regions corresponding to the surface region among the regions in the plurality of photographed images photographed by the plurality of photographing units. The image processing apparatus according to claim 1, wherein the area of the image projected onto the surface area is determined based on the parameters of   The determination unit is an imaging unit configured to capture the plurality of areas from a plurality of areas corresponding to the surface area among the plurality of areas in the plurality of captured images captured by the plurality of imaging sections, and the reference imaging section The image processing apparatus according to claim 5, wherein the area of the image projected onto the surface area is determined based on the coordinates of.   The determination unit is configured to determine, from among a plurality of areas in the plurality of photographed images photographed by the plurality of photographing units, photographing directions of the respective photographing units for photographing the plurality of areas from the plurality of areas corresponding to the surface area. The image processing apparatus according to claim 5, wherein the area of the image projected on the surface area is determined based on a photographing direction of a reference photographing unit.   The image processing apparatus according to any one of claims 5 to 7, wherein the reference photographing unit is a photographing unit having the widest field of view among the plurality of photographing units.   The image processing apparatus according to any one of claims 5 to 7, wherein the reference photographing unit is a photographing unit virtually arranged at a designated position. The image capturing apparatus further includes acquisition means for acquiring an image of the area determined by the determination means included in the plurality of photographed images photographed by the plurality of photographing units from the plurality of photographing units.
The image processing apparatus according to any one of claims 1 to 9, wherein the projection unit projects the image acquired by the acquisition unit onto the surface area. The image processing apparatus further comprises selection means for selecting an image used to generate the virtual viewpoint image from the photographed images photographed by the plurality of photographing units.
11. The image processing apparatus according to claim 1, wherein the determination unit determines the region of the image projected onto the surface region from the region corresponding to the surface region among the regions in the image selected by the selection unit. Image processing apparatus as described.   The determination unit is configured to set an area of an image projected on the surface area from an area included in a captured image captured by an imaging section other than the imaging section in which a predetermined defect is detected among the plurality of imaging sections. The image processing apparatus according to any one of claims 1 to 11, which determines. An information processing method executed by the image processing apparatus;
As the image projected on the surface area which is an area existing on the surface of the background model which is a model of the background of the virtual viewpoint image generated based on the plurality of photographed images photographed by the plurality of photographing units, the plurality of photographings A determination step of determining which area of the captured image included in the plurality of captured images captured by the unit is to be projected;
Projecting the image of the area determined in the determining step included in the plurality of photographed images photographed by the plurality of photographing units onto the surface area;
Information processing method including:   The program for functioning a computer as each means of the image processing apparatus in any one of Claims 1-12.

Images