JP7233881B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to a technique for generating an image from an arbitrary virtual viewpoint based on multiple viewpoint images captured from multiple viewpoints.

2. Description of the Related Art There is a technique of generating a virtual viewpoint image at an arbitrary virtual viewpoint using images of multiple viewpoints captured at multiple viewpoints. For example, by using a virtual viewpoint image, the highlight scenes of soccer or basketball can be viewed and viewed from various angles, so that it is possible to give the user a high sense of realism compared to normal images.

To generate and browse virtual viewpoint images, images captured by multiple cameras are aggregated in an image processing device such as a server, the foreground and background are separated, a 3D model is generated, and processing such as rendering is performed. It is realized by transmitting viewpoint images. At this time, model data of a stadium or the like prepared in advance is used as the background of the virtual viewpoint image, the shape of the foreground is estimated, and a three-dimensional shape model is generated by coloring them.

In order to improve the accuracy of the shape estimation of the foreground, that is, the image quality of the foreground, the angle of view of each camera is narrowed and optically enlarged with the focus and orientation adjusted to the point of interest. A plurality of gaze points are provided according to the target location.

On the other hand, since the background stadium is reflected in the image near the point of interest, it is necessary to widen the angle of view of each camera and capture the entire stadium in an optically sharpened image regardless of the point of interest.

Japanese Patent Application Laid-Open No. 2002-200000 describes a technique in which a plurality of projectors superimpose and display images on a model on which the images are projected.

JP 2008-70397 A

However, since the angle of view required for the camera to generate the foreground image and the background image differs, an appropriate background image is generated based on the image captured by the camera used to generate the foreground image. Sometimes it was not possible.

With the technique of Patent Document 1, it was not possible to synchronously project images of a group of cameras that have common world coordinate parameters and that have been synchronously photographed onto a model that also has common world coordinates.

An object of the present invention is to generate an appropriate background image based on an image captured by a camera used to generate a foreground image.

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that generates a virtual viewpoint image, and is photographed by a first camera group composed of one or more cameras installed facing a first position. a first foreground generating means for generating a foreground image based on the obtained image; and a second camera group composed of one or more cameras installed facing a second position different from the first position. a second foreground generating means for generating a foreground image based on the image captured by the first and second camera groups; a texture image for the background model generated based on the image captured by the first and second camera groups; Background generation means for generating a background image of the virtual viewpoint image based on the background model, and one or more cameras installed facing a third position different from the first and second positions. and a third camera group, wherein the background generating means generates an image captured by the third camera group in addition to the images captured by the first and second camera groups The background image is generated further based on the texture image for the background model generated based on .

According to the present invention, an appropriate background image can be generated based on an image captured by a camera used for generating a foreground image.

Block diagram showing a configuration example of an image processing system Flowchart showing an example of a processing procedure for generating a virtual viewpoint image A diagram showing a schematic configuration of an imaging system Diagram for explaining the shooting range by the foreground generation system Diagram for explaining the shooting range by another foreground generation system Diagram for explaining the range of the background image Schematic diagram showing how a virtual viewpoint image is rendered Schematic diagram showing the relationship between the shooting range and the virtual viewpoint rendering range by two foreground generation systems Block diagram showing a configuration example of an image processing system Schematic diagram showing the imaging range of the two foreground generation systems Block diagram showing a configuration example of an image processing system Flowchart showing a procedure example of virtual viewpoint image generation processing Schematic diagram showing the imaging range of the imaging system and the pixel positions and non-imaging areas Diagram showing a hardware configuration example of the foreground generation server

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the constituent elements described in this embodiment are merely examples, and are not intended to limit the scope of the present invention. In addition, not all combinations of the components described in the embodiments are essential to the means for solving the problems, and within the scope of the gist of the invention described in the claims, Various modifications and alterations are possible. In this embodiment, an example in which the background image is used to generate the virtual viewpoint image will be described, but the background image may not necessarily be used to generate the virtual viewpoint image. For this reason, it is not bound by relative geometric installation conditions between multiple cameras.

In this specification, alphabetical letters at the end of reference numerals are used to identify individual configurations in similar configurations. When description is made by omitting the alphabet at the end, it is assumed that each configuration is described without particular distinction.

[Embodiment 1]
In this embodiment, a form of generating a background image used for generating a virtual viewpoint image will be described.
FIG. 1 is a block diagram showing an example of the overall configuration of an image processing system according to this embodiment.

An image processing system (virtual viewpoint image generation system) 100 has foreground generation systems 110A and 110B, a background generation system 120, and a rendering system 130, and generates a virtual viewpoint image based on an image captured by a camera. The foreground generation systems 110A and 110B, the background generation system 120, and the rendering system 130 are each connected by a network such as Ethrenet (registered trademark). Although the present embodiment has two foreground generation systems 110A and 110B, the present embodiment is not limited to this, and may be configured to have two or more foreground generation systems 110. is also possible.

Foreground generation systems 110A and 110B are systems required to generate foreground images of virtual viewpoint images. The foreground generation system 110A and the foreground generation system 110B have the same configuration. Henceforth, the foreground generation system 110A will be described, and the description of the foreground generation system 110B will be omitted. The foreground generation system 110A has imaging systems 111a to 111z and a foreground generation server 113 inside. The foreground generation systems 110A and 110B are configured for each gaze point with the orientation and focal length of the camera adjusted. However, in this embodiment, the gazing points of a plurality of cameras may not match and may be shifted.

The photographing system 111 realizes a function of photographing an object for generating a virtual viewpoint image with a camera and transferring a digital image obtained by performing conversion to each block. In order to generate a virtual viewpoint image, the foreground generation system 110 includes a plurality of photographing systems 111 having a similar configuration, and in this embodiment, 26 systems 111a to 111z are configured. However, the present embodiment is not limited to this, and as long as a necessary and sufficient number of imaging systems are provided to obtain the desired quality of the foreground image of the virtual viewpoint image, the number of imaging systems is less than or greater than 26. number of units. Each imaging system 111 has an imaging unit 112 therein.

The imaging unit 112 has a function of converting an image captured by a camera into a digital image. A camera included in the imaging unit 112 can receive a synchronization signal, and is synchronized by a camera control unit 117, which will be described later. Although the imaging systems 111a to 111z have similar functions, different cameras and lenses may be used for the respective cameras.

The foreground generation server 113 has a foreground image generation unit 114 , a parameter acquisition unit 115 , a foreground shape estimation unit 116 and a camera control unit 117 . The foreground generation server 113 estimates the shape of the foreground, generates foreground model data, and transmits the generated foreground model data to the rendering system 130 . The foreground model data is data representing the three-dimensional shape of the foreground.

The foreground image generation unit 114 separates a foreground image and a background image from the digital image data captured and converted by the imaging systems 111a to 111z, and generates foreground image data obtained by extracting only the foreground image that is the main subject (object). Generate. The generated foreground image data is transmitted to foreground shape estimation section 116 . The foreground image is, for example, an image obtained by extracting a moving person (moving body) such as a player from image data. An object may be an object such as a ball. The background image is, for example, an image obtained by extracting an object such as lawn that remains stationary or nearly stationary. Note that the method for separating the foreground image and the background image is not limited to the background subtraction method, and any known method can be applied. A stationary object refers to an object that continues to be stationary or almost stationary.

The parameter acquisition unit 115 holds the camera parameters of the camera of the imaging unit 112 (hereinafter also referred to as the camera of the imaging system 111) acquired by calibration in advance, and provides them to the foreground shape estimation unit 116 . The camera parameters consist of extrinsic parameters representing the position and orientation of the camera, and internal parameters representing the pixel pitch, image center, and the like. The camera parameters may also include distortion parameters representing distortion of the image due to the lens.

The foreground shape estimation unit 116 estimates at least the three-dimensional shape of the foreground using the camera parameters of the imaging systems 111a to 111z acquired from the parameter acquisition unit 115 and the foreground image data received from the foreground image generation unit 114, and estimates the foreground model. Generate data. Methods for estimating the three-dimensional shape of the foreground include, for example, a method called Visual Hull, but the present embodiment is not limited to this, and any method for estimating the shape of the foreground may be used. The generated foreground model data is sent to the rendering system 130 .

The camera control unit 117 transmits a signal for synchronizing the shooting timings of the cameras of the shooting systems 111a to 111z to the shooting systems, synchronizes the shooting timings of the cameras of the shooting systems 111a to 111z, and instructs the start and stop of shooting. . Signals indicating the start and stop states of imaging are transmitted to the control unit 134, which will be described later. Note that this embodiment is not limited to this, and a separate signal line for a synchronization signal may be provided, and synchronization signal control may be performed using radio or the like.

The foreground generation system 110B is a system having the same configuration and functions as the foreground generation system 110A. The foreground generation system 110B does not need to have the same number of shooting systems 111 as the foreground generation system 110A. Just do it.

The background generation system 120 has a parameter acquisition unit 121 , a model acquisition unit 122 , a background image generation unit 123 , a composition determination unit 124 and a texture generation unit 125 . The background generation system 120 uses digital image data captured by the imaging system 111 to generate a background texture image to be pasted on the background model. The background model is data representing the three-dimensional shape of the background.

The parameter acquisition unit 121 has the same function as the parameter acquisition unit 115 , holds the camera parameters of the camera of the imaging system 111 acquired in advance by calibration, and provides the texture generation unit 125 with the camera parameters.

The model acquisition unit 122 acquires background mesh model data obtained by converting the background into a mesh model as a background model, and provides the texture generation unit 125 with the background model data. For example, when shooting a sport such as baseball or soccer, mesh model data of a stadium where the sport is actually held can be considered. may be used. A mesh model is data such as CAD data expressed by a set of surfaces of a three-dimensional spatial shape. The format for expressing the background model is not limited to the mesh model format, and may be other known formats such as point cloud and voxel.

The background image generation unit 123 separates the foreground image and the background image from the digital image data captured by the imaging systems 111a to 111z, and generates background image data by extracting only the background image. The generated background image data is sent to the texture generation unit 125 .

The synthesis determination unit 124 recognizes the foreground generation system 110 existing within the image processing system 100 . Then, the synthesis determination unit 124 receives an instruction from the user as to whether or not to use the image data of which foreground generation system for the image data received from each foreground generation system 110 , and provides the information to the texture generation unit 125 . do.

Using the background image data, the texture generation unit 125 performs projective transformation on the meshes of the background mesh model data using the camera parameters, and performs projection and synthesis processing. The background image data is obtained from the digital image data to be used, among the digital image data obtained by the foreground generation systems 110A and 110B, based on the information about whether or not the image data can be used, which is provided by the composition determination unit 124. 123 generated data. The background mesh model data is data provided from the model acquisition unit 122 . Camera parameters are provided from the parameter acquisition unit 121 .

Then, the texture generation unit 125 generates a texture image of each mesh by projection onto the background mesh model and synthesis. That is, the texture generator 125 uses the digital image data obtained by the foreground generation systems 110A and 110B to generate a background texture image for the background mesh model data. The texture generation unit 125 transmits the generated background texture image together with the background model data to the rendering system 130 . A texture is an image pasted to express the texture of the surface of an object.

As a method of projecting the camera image onto the background mesh model, for example, the following method is conceivable. Four points on a certain mesh, for example, if the mesh is a quadrilateral, it is linked to the edge of the texture image, and from the correspondence between the coordinates of the four points on the camera image and the vertex coordinates of the texture image, the homography is obtained and perspective is obtained. A method such as conversion is conceivable. When the camera images are overlapped as a result of perspective transformation of the camera images and projection onto the background mesh model, they are combined to generate an image. Note that the present embodiment is not limited to this, and any method of projecting the background image data onto the texture image may be used.

The rendering system 130 has a rendering section 131 , an operation section 132 , an output section 133 and a control section 134 . The rendering system 130 uses the foreground model data acquired from the foreground generation system 110A or 110B, the background model data acquired from the background generation system 120, and the background texture image to generate a virtual viewpoint image according to user's operation.

The rendering unit 131 uses the foreground model data received from the foreground generation systems 110A and 110B and the background model data and background texture image received from the background generation system 120 to render an image from a virtual viewpoint by computation. The computation includes, for example, computation by a CPU or GPU. Further, an operation for changing the virtual viewpoint image is performed by receiving an operation from the user through the operation unit 132, which will be described later.

The operation unit 132 has operation members such as a keyboard, a mouse, a joystick and a touch pad, or input means such as file input, and is used by the user to change the virtual viewpoint, instruct to start shooting, and other instructions to each unit of the image processing system 100 . accept. Information on the changed virtual viewpoint is transmitted to the rendering unit 131 .

The output unit 133 has an image signal output terminal such as a display port or SDI, and a display device such as a TV, a display, or a projector for inputting the image signal and displaying it as an image. The output unit 133 outputs the rendered virtual viewpoint image as an image that the user can visually confirm. Note that the present embodiment is not limited to this, and the virtual viewpoint image may be output as image data as a still image or moving image, and may be distributed to a network or stored in a storage device.

The control unit 134 communicates with the foreground generation systems 110A and 110B and the background generation system 120 to perform various controls. The control unit 134 starts and stops generation of foreground image data and foreground model data by the foreground generation systems 110A and 110B, starts and stops generation of background image data and background texture images by the background generation system 120, and performs gaze point switching operations. etc. That is, the control unit 134 controls each block of the image processing system 100 .

FIG. 2 is a flowchart illustrating an example of a virtual viewpoint image generation processing procedure by the image processing system 100 . The virtual viewpoint image generation processing of the image processing system of this embodiment will be described using this flowchart.

FIG. 2A is a flow chart showing an example of the foreground model data generation processing procedure by the foreground generation system 110A. Although processing by the foreground generation system 110A will be described here, the same processing as in FIG. 2A is also performed in the foreground generation system 110B. Each symbol S below means a step in the flow chart. These are the same for FIG. 2 and subsequent figures.

The camera control unit 117 of the foreground generation system 110A receives the signal of the shooting start instruction transmitted from the control unit 134 in S221 described later (S201).

FIG. 3 is a diagram showing a schematic configuration of the imaging system. As shown in FIG. 3, the imaging systems 111a to 111z are installed so as to surround a stadium 300. As shown in FIG. A stadium 300 shows the background mesh model data of this embodiment, and uses a plane model of only the ground without spectator seats. A plane model is a three-dimensional model represented as a plane in a three-dimensional space. Although the present embodiment shows the case of using a plane model, the present embodiment is not limited to this, and a model having a three-dimensional structure may be used.

A gaze point 301 is the gaze point of the foreground generation system 110A. The camera of each photographing system 111a to 111z is directed toward the point of interest 301 and installed so that the point of interest 301 is in focus. That is, the gaze point (shooting center) of the cameras of the imaging systems 111a to 111z is the gaze point 301. FIG.

The camera control unit 117 synchronizes the shooting timings of the cameras of the shooting systems 111a to 111z (S202).

When the imaging timings of the cameras of the imaging systems 111a to 111z are synchronized by the camera control unit 117, the camera control unit 117 instructs the imaging systems 111a to 111z to start imaging. The imaging units 112 of the imaging systems 111 a to 111 z convert images captured by cameras into digital image data, and transmit the digital image data to the foreground image generation unit 114 and the background image generation unit 123 . Further, when the synchronous photographing is started, the camera control unit 117 transmits a signal indicating that the synchronous photographing is started to the control unit 134 (S203).

The foreground image generating unit 114 separates the foreground image and the background image from the digital image data obtained by converting the image captured by the imaging unit 112 by the imaging system 111a to 111z with the camera, removes the background image, and extracts only the foreground image. Generate image data. The foreground image generation unit 114 then provides the generated foreground image data to the foreground shape estimation unit 116 (S204).

The foreground shape estimation unit 116 estimates the shape of the foreground using the foreground image data received from the foreground image generation unit 114 and the camera parameters acquired from the parameter acquisition unit 115, and generates foreground model data (S205). It is assumed that the parameter acquisition unit 115 has camera parameters acquired in advance by calibration before this flow.

The foreground shape estimation unit 116 transmits the generated foreground model data to the rendering unit 131 (S206).

FIG. 2B is a flow chart showing an example of the background data generation processing procedure by the background generation system 120 .
The composition determination unit 124 of the background generation system 120 acquires which image data of the foreground generation system 110 preset by the user is used to generate the background texture image (S211). In this embodiment, background image data is generated using all image data of foreground generation systems 110A and 110B.

The background image generation unit 123 receives the digital image data captured and converted in S203 from the foreground generation system 110A or 110B based on the determination result of the composition determination unit 124 acquired in S211 (S212). For example, assume here that image data has been received from the foreground generation system 110A.

Synthesis determination unit 124 determines whether background image generation unit 123 has acquired data from all foreground generation systems 110A and 110B set as data for generating background image data (S213).

Synthesis determination unit 124 acquires data from all of foreground generation systems 110A and 110B, and when determining that all data in the synthesis range is complete, transitions the process to S214. If the composition determination unit 124 determines that data has not been obtained from all of the foreground generation systems 110A and 110B and all the data in the composition range has not been obtained, the process returns to S212, and the processes of S212 and S213 are repeated. . For example, in S212, the image data of the foreground generation system 110A has been received, but the image data of the foreground generation system 110B has not been received. become. After that, when the reception of the image data of the foreground generation system 110B is completed in S212, the composition determination unit 124 determines in S213 that the image data of all the foreground generation systems set in the composition determination unit 124 is complete, and performs the processing. to S214.

The background image generation unit 123 separates the digital image data received in S212 into a foreground image and a background image, and generates background image data by extracting only the background image. The background image generation unit 123 then provides the generated background image data to the texture generation unit 125 (S214).

The texture generation unit 125 projects the background image data received from the model acquisition unit 122 by projectively transforming the background model data acquired from the model acquisition unit 122, and synthesizes them to generate a background texture image. (S215). Note that the texture generation unit 125 acquires camera parameters from the parameter acquisition unit 121 . It is assumed that the parameter acquisition unit 121 acquires and holds camera parameters in advance through calibration, and acquires the camera parameters before this flow. It is also assumed that the model acquisition unit 122 has acquired the background model data before this flow.

FIGS. 4A and 4B are diagrams schematically showing the shooting ranges obtained by the shooting systems 111a to 111z of the foreground generation system 110A.

FIG. 4A is a diagram showing a range photographed by the photographing system 111t. When the photographing system 111t takes a picture from the position shown in the drawing while focusing on the point of regard 301, the photographing range of the photographing system 111t becomes a range indicated by an area 400 with the point of regard 301 as the photographing center. . That is, when the imaging system 111t is focused on the gaze point 301, the imaging range of the imaging system 111t becomes the range indicated by the area 400. FIG. In this way, when all the 26 imaging ranges of the imaging systems 111a to 111z photographed by focusing on the gaze point 301 from the illustrated position are combined, as shown in FIG. 4(b), the total imaging range is A first area 401 is obtained with the point of gaze 301 as the imaging center. A first area 401 covers approximately the right half of the stadium 300 . That is, the background image generator 123 can generate the background image data corresponding to the first area 401 using the digital image data received from the foreground generation system 110A.

Similarly, FIG. 5 is a diagram schematically showing the shooting range obtained by the shooting systems 111A to 111Z of the foreground generation system 110B. Point of regard 501 is the point of regard for foreground generation system 110B. When the photographing systems 111A to 111Z of the foreground generation system 110B focus on the point of interest 501 and photograph from the illustrated position, the total photographing range of the photographing systems 111A to 111Z is centered on the point of interest 501. It becomes the second area 502 . A second region 502 covers approximately the left half of the stadium 300 . That is, the background image generator 123 can generate background image data corresponding to the second area 502 using the digital image data received from the foreground generation system 110B.

In the present embodiment, the composition determination unit 124 determines that both the foreground generation system 110A and the foreground generation system 110B are used, so background image data is generated by combining them.

FIG. 6 is a schematic diagram showing the range of the background image generated by the background generation system 120. As shown in FIG. When the imaging systems of the foreground generation systems 110A and 110B are focused on the gazing points 301 and 501 for imaging, the combined imaging range of the imaging systems is the first area 401 in FIG. A total area 601 including the second area 502 is obtained. A total area 601 covers the entire stadium 300 . That is, the background generation system 120 can generate background image data corresponding to the entire area 601 covering the entire stadium 300 .

The texture generation unit 125 performs projective transformation on the background image data generated by the background image generation unit 123 using the camera parameters acquired from the parameter acquisition unit 121, and projects the background image data onto the stadium 300, which is the background model data, to generate a background texture. Generate an image. The texture generation unit 125 then transmits the background model data and the generated background texture image to the rendering unit 131 (S216).

FIG. 2(c) is a flowchart showing an example of a processing procedure for generating a virtual viewpoint image by the rendering system 130. As shown in FIG.
When the operation unit 132 is operated by the user and receives an operation to start generating a virtual viewpoint image, it notifies each unit of the image processing system 100 that the operation to start generating a virtual viewpoint image has been received, and generates a virtual viewpoint image. (S221).

When the control unit 134 of the rendering system 130 receives notification from the operation unit 132 that the operation to start generating the virtual viewpoint image has been received, it instructs the camera control units 117 of the foreground generation systems 110A and 110B to start synchronous shooting. (S222). When the synchronous photographing is started, the control unit 134 receives from the camera control unit 117 the signal indicating that the synchronous photographing has been started, which was transmitted in S203.

The rendering unit 131 receives the foreground model data generated in S206 from the foreground shape estimation units 116 of the foreground generation systems 110A and 110B (S223).

The rendering unit 131 receives the background model data and the background texture image generated in S216 from the texture generation unit 125 (S224).

The operation unit 132 receives an operation of the virtual viewpoint by the user, that is, an operation of changing the virtual viewpoint coordinates, and transmits the changed virtual viewpoint coordinate information to the rendering unit 131 (S225).

The rendering unit 131 uses the received foreground model data and background model data, and uses the virtual viewpoint coordinates received from the operation unit 132 to render the image from the virtual viewpoint by calculation of the CPU, GPU, or the like (S226).

FIG. 7 is a schematic diagram showing how a virtual viewpoint image is rendered.
A virtual viewpoint 701 represents a virtual viewpoint camera, which is an arbitrary viewpoint at which no camera is installed. A virtual viewpoint rendering range 702 indicates a range rendered as a virtual viewpoint image from the virtual viewpoint 701 .

The virtual viewpoint rendering range 702 has a non-shooting area 703 that exceeds the range of the first area 401, which is the shooting range of the foreground generation system 110A. The non-imaging area 703 is an area filled with black. For the non-shooting area 703, the background texture image cannot be generated using only the image data of the foreground generation system 110A.

FIG. 8 is a schematic diagram showing the relationship between a total area 601 obtained by combining the imaging ranges of the foreground generation systems 110A and 110B and a virtual viewpoint rendering range 702. As shown in FIG. In this embodiment, both the image data of the foreground generation systems 110A and 110B are used, so the background texture image is generated within the range of the entire area 601. FIG. Therefore, the entire area 601 includes the entire stadium 300 in the shooting range, and a background texture image without gaps can be generated in the range visible from the virtual viewpoint rendering range 702 .

The image generated by the rendering unit 131 is sent to the output unit 133 and viewed by the user through a display device (not shown) such as a display, or acquired by the user as image data such as a file (S227).

As described above, according to this embodiment, an appropriate background image can be generated based on an image captured by a camera with a narrow angle of view. Specifically, it is possible to widen the background generation range of the virtual viewpoint image by using a camera with a limited shooting range facing the gaze point. That is, when estimating the shape of the foreground for each of a plurality of fixation points, the range of background generation can be expanded without adding a new camera.

(Modification)
Although the present embodiment has been described above, the present invention is not limited to this.
For example, in this embodiment, the background generation system 120 and the rendering system 130 are separate systems, but they may be the same system.

Having a user interface (selection means) for selecting a gaze point, the rendering system 130 may generate a virtual viewpoint image using foreground model data generated by a foreground generation server that selects the gaze point.

Also, in the present embodiment, two image data of the foreground generation system 110A and the foreground generation system 110B are sent to the background generation system 120, but the present invention is not limited to this. For example, the user interface of operation unit 132 may allow each of foreground generation systems 110A and 110B to turn on or off the transfer of image data to background generation system 120. FIG. In that case, the background generation system 120 generates a background texture image using the image data of the foreground generation system 110 whose image data transfer is turned on.

In addition, the image processing system further includes a third imaging system having a third camera group composed of one or more cameras installed toward a gaze point different from the gaze points 301 and 501 of the imaging system 111. may The third camera group preferably captures an image with a wider angle of view than the images captured by the cameras (first and second camera groups) of the imaging system 111 of the foreground generation systems 110A and 110B. It is more preferable to photograph the whole. Note that when the third camera group captures the entire stadium 300 , it is preferable to set the gazing point of the cameras of the third camera group substantially at the center of the stadium 300 . In that case, the texture generation unit 125 further uses images captured by the third camera group of the third imaging system in addition to the images captured by the camera groups of the foreground generation systems 110A and 110B to create the background model. Generate a background texture image to paste. Also, the texture generation unit 125 renders the image captured by the third camera group with a higher weight than the images captured by the first and second camera groups of the foreground generation systems 110A and 110B. An image may be generated. This makes it possible to generate a more appropriate background image than when based only on the images captured by the first and second camera groups of the foreground generation systems 110A and 110B.

Further, the image capturing system 111 has a distortion correction unit that corrects image distortion using a distortion parameter in the foreground generation server 113 or the background generation system 120, and corrects captured image data by the distortion correction unit. good too. Also, the distortion correction unit may be included in the imaging system 111 .

Further, in the present embodiment, image data is transmitted from the photographing system 111 to the foreground generation server 113 and the background generation system 120. However, if the image processing system 100 has an aggregation server and the aggregation server aggregates the data, may be in any form. Then, the aggregation server (sorting means) may sort the data aggregated by the aggregation server to the foreground generation server 113 of the foreground generation system 110A, the foreground generation server 113 of the foreground generation system 110B, and the background generation system 120. . Specifically, the aggregating server may distribute images of the imaging system 111 of the foreground generation system 110A to the foreground generation server 113 of the foreground generation system 110A. The aggregating server may distribute images of the imaging system 111 of the foreground generation system 110B to the foreground generation server 113 of the foreground generation system 110B. The aggregating server may distribute the image of the imaging system 111 of the foreground generation system 110A and the image of the imaging system 111 of the foreground generation system 110B to the background generation system 120 .

Further, the image processing system 100 may include a database system having a storage, and transmit and receive necessary data to and from the foreground generation system 110, background generation system 120, and rendering system 130 via the database. The storage is large enough to store camera parameters, background model data, foreground image data generated by foreground generation system 110, background image data and foreground model data, background texture images generated by background generation system 120, and the like. It is a capacitive storage device.

[Embodiment 2]
An image processing system according to this embodiment will be described with reference to FIGS. 9 and 10. FIG.
In this embodiment, synthesis of background image data will be described. It should be noted that, in this embodiment, the same reference numerals are given to the same devices as in the image processing system of the above-described embodiment, and the description thereof will be omitted.

FIG. 9 is a block diagram showing a configuration example of the image processing system of this embodiment.
An image processing system (virtual viewpoint image generation system) 900 of this embodiment includes a foreground generation system 110A, a foreground generation system 110B, a background generation system 120, and a rendering system 130, as in the above embodiments.

The background generation system 120 has a weighting determination unit 901 in addition to the parameter acquisition unit 121 , the model acquisition unit 122 , the background image generation unit 123 , the composition determination unit 124 , the texture generation unit 125 .

When the texture generation unit 125 projects the camera image (background image data) onto the background mesh model by projective transformation, the weight determination unit 901 applies a weight for synthesizing the camera image (background image data) projected onto the mesh model. to derive the That is, the weighting determination unit 901 derives the weighting of each camera image with respect to overlapping regions in the camera images projected onto the mesh model. The weighting process by the weighting determination unit 901 is executed when the background texture image is generated in S216.

Weighting processing by the weighting determination unit 901 will be described with reference to FIG.
FIG. 10 is a schematic diagram showing the shooting ranges of the foreground generation systems 110A and 110B.

An overlapping area 1001 is an area where the first area 401, which is the imaging range of the foreground generation system 110A, and the second area 502, which is the imaging range of the foreground generation system 110B, overlap. In other words, the overlapping area 1001 is a range in which background image data can be generated using either of the foreground generation systems 110A and 110B.

A pixel 1002 indicates a point in an overlapping area 1001 where the first area 401 and the second area 502 overlap on the mesh model serving as the background.

Assume that the distance 1003A between the pixel 1002 and the point of interest 301 is La, and the distance 1003B between the pixel 1002 and the point of interest 501 is Lb.

At this time, regarding the pixel 1002, the numerical value of the pixel value X obtained by synthesizing the pixel value Xa of the background image data generated by the foreground generation system 110A and the pixel value Xb of the background image data generated by the foreground generation system 110B is Derived by calculation. As a method of deriving the pixel value X, for example, the following formula is used.

By using the above formula, it is possible to weight pixels closer to the point of gaze. In the case of the pixel 1002, the length Lb is smaller than the length La, and the weighting of the pixel value Xb is greater than the weighting of the pixel value Xa.

By weighting using the distance from the gaze point in this way, it is possible to gradually increase the weight so that the boundary line does not stand out in the switching portion of the camera image (digital image data by the foreground generation system).

In the present embodiment, the pixel value X is derived from the above formula (1), but the method of deriving the pixel value X is not limited to this. For example, it may be a value obtained by normalizing the range of formula (1). Further, the weighting for combining the pixel value Xa and the pixel value Xb when obtaining the pixel value X is obtained based on the distance 1003A and the distance 1003B, and weighting is performed based on the smaller value of the length La or the length Lb. should be derived so as to be heavy.
As for weights for synthesis, a method of simply using an average value or an intermediate value may be used.

Alternatively, the image quality of the digital image data acquired from each foreground generation system 110 may be evaluated, the evaluation of high resolution and high contrast may be quantified, and weighting may be performed based on the numerical values.

In addition, if there are pixels that fail to be captured in part or all of the imaging range captured by the foreground generation system 110, the weight of those pixels may be set to 0 or a numerical value close to 0.

As described above, according to the present embodiment, when there is an overlapping area in camera images (digital image data) obtained by a plurality of imaging systems, weighting processing is performed so that the boundary line between camera images to be switched is generation can be suppressed.

[Embodiment 3]
An image processing system according to this embodiment will be described with reference to FIGS. 11 and 12. FIG.
In this embodiment, a method of designating a generation range in generating background image data will be described. It should be noted that, in this embodiment, the same reference numerals are given to the same devices as in the image processing system of the above-described embodiment, and the description thereof will be omitted.

FIG. 11 is a block diagram showing a configuration example of the image processing system of this embodiment.
An image processing system (virtual viewpoint image generation system) 1100 of this embodiment includes a foreground generation system 110A, a foreground generation system 110B, a background generation system 120, and a rendering system 130, as in the above embodiments.

The imaging system 111a of the foreground generation system 110A has a foreground/background separating unit (hereinafter referred to as a separating unit) 1101 in addition to the imaging unit 112 .

The background generation system 120 includes a parameter acquisition unit 121 , a model acquisition unit 122 , a synthesis determination unit 124 , a texture generation unit 125 , and a range designation unit (hereinafter referred to as designation unit) 1102 .

A separating unit 1101 separates a foreground image and a background image from digital image data obtained by converting an image captured by a camera in the imaging unit 112 to generate foreground image data and background image data. At this time, the background image generation unit 123 does not generate background image data for the range designated by the designation unit 1102, which will be described later.

A designating unit 1102 designates the range of background image data generated by the separating unit 1101, and receives the background image data of the designated range. The received background image data is sent to the texture generator 125 .

FIG. 12 is a flowchart of virtual viewpoint image generation processing by the image processing system 1100 . FIG. 12A shows a flow of an example of the foreground data generation processing procedure by the foreground generation system 110A, and FIG. 12B shows a flow of an example of the background data generation processing procedure by the background generation system 120A. Although processing by the foreground generation system 110A will be described here, the same processing as in FIG. 12A is also performed in the foreground generation system 110B. A flow showing an example of the virtual viewpoint image generation processing procedure by the rendering system 130 is the same as the flow shown in FIG.

A separating unit 1101 separates digital image data (captured data) acquired from the imaging unit 112 into a foreground image and a background image, generates foreground image data by extracting only the foreground image, and performs foreground shape estimation on the generated foreground image data. It is provided to the unit 116 (S1201).

The separating unit 1101 separates the captured data acquired from the imaging unit 112 into a foreground image and a background image, and generates background image data according to the range specified by the specifying unit 1102 in S1211 (described later) (S1202). The generated background image data is transmitted to the designation unit 1102 .

The designating unit 1102 determines which of the foreground generation systems 110A and 110B is used to generate the overlap region, according to the use setting of the foreground generation systems 110A and 110B set in S211 (S1211). For example, the designation unit 102 selects one of the foreground generation systems 110A and 110B according to the position and direction of the virtual viewpoint.

A method of specifying a range in the background image data by the specifying unit 1102 will be described using the overlapping area 1001 in FIG. 10 .

The overlapping area 1001 is an area where the background image data can be generated by either one of the foreground generation systems 110A and 110B, because the background image data can be generated by either of them. As in the second embodiment, when weighting determination unit 901 performs weighting calculation for pixel 1002, the weight for the image obtained by the camera corresponding to foreground generation system 110A is Lb/(La+Lb). The weight for the image obtained by the camera corresponding to the foreground generation system 110B is La/(La+Lb). At this time, since La>Lb, it can be seen that the image obtained by the foreground generation system 110B is weighted more heavily than the image obtained by the foreground generation system 110A. The designation unit 1102 performs weighted selection in this way.

When the foreground generation system 110B is selected, the designation unit 1102 designates to the separation unit 1101 as unnecessary pixel values the locations corresponding to the pixels 1002 of the imaging systems 111a to 111z of the foreground generation system 110A.

FIG. 13 is a diagram showing how the specifying unit 1102 specifies the generation range of the background image data for the imaging system 111t. FIG. 13A shows the relationship between the imaging range of the imaging system 111t and the positions of the pixels, and FIG. 13B shows the relationship between the imaging range of the imaging system 111t and the non-image generation range.

As shown in FIG. 13A, the pixel 1002 is positioned within a range indicated by an area 400, which is the shooting range of the stadium 300 shot by the shooting system 111t whose shooting center is set to the gaze point 301. As shown in FIG. As shown in FIG. 13A, pixels 1002 included in the imaging range area 400 are excluded from the background image data generation range generated by the separation unit 1101 .

When the same processing is performed for each pixel in the overlapping area 1001, the background image data is not generated for the area 1301 shown in the non-image generation range in FIG. The transfer size of image data can be reduced.

The specifying unit 1102 receives the background image data from the separating unit 1101 (S1212). For example, here, background image data is received from the separating unit 1101 of the imaging system 111a.

The designating unit 1102 determines whether or not the image data received from the separating unit 1101 has received all the data of the combining range (S1213). That is, the designating unit 1102 determines whether or not data has been acquired from all of the foreground generation systems 110A and 110B set as data for generating background image data in the composition determination unit 124 . When the specifying unit 1102 determines that all data have been received (YES in S1213), the process proceeds to S1214. If the designation unit 1102 determines that all the data has not been received (NO in S1213), the process returns to S1212, and the processes of S1212 and S1213 are repeated. For example, if the digital image data of the imaging system 111a is received in S1212, but the other digital image data is not received, the process returns to S1212 and waits until the digital image data is received. . After that, when reception of the digital image data of the foreground generation system from all the remaining photographing systems of the foreground generation system is completed in S1212, the designating unit 1102 receives all the background image data set in the synthesis determination unit 124 in S1213. I judge. Based on this determination result, the specifying unit 1102 shifts the processing to S1214.

The texture generation unit 125 projects the background image data acquired from the specification unit 1102 by projectively transforming the background model data acquired from the model acquisition unit 122 using the camera parameters acquired from the parameter acquisition unit 121 . A background texture image is generated by synthesizing the projected images (S1214).

Although the generation range of the background image data of the overlapping area has been described above, the present embodiment is not limited to this.

The highest weighting may be selected according to other weighting priority methods previously described.
When a point of interest is selected by a user operation, the shooting range of the point of interest selected in the overlapping area may be entirely selected, and other overlapping points of interest may not be used. Alternatively, the user may specify the priority of the gaze point and specify the generation range of the background image data according to the specified priority.

Next, the hardware configuration of each device constituting this embodiment will be described in more detail. As described above, in the present embodiment, the foreground generation server 113 implements hardware such as FPGA and/or ASIC, and an example in which the above-described processes are executed by such hardware has been described. The same applies to the shooting system 111, the background generation system 120, and the rendering system 130 within the foreground generation system 110. FIG. However, at least one of the above devices may use, for example, a CPU, DSP, etc., and execute the processing of the present embodiment by software processing.

FIG. 14 is a diagram showing a hardware configuration example of the foreground generation server 113 for realizing the functional configuration shown in FIG. 1 by software processing. Devices such as the imaging system 111, the background generation system 120, and the rendering system 130 can also have the hardware configuration shown in FIG. The foreground generation server 113 has a CPU 1131 , a RAM 1132 , a ROM 1133 , a secondary storage device 1134 and an input/output interface 1135 . These components are connected by a bus, allowing data to be sent and received between the components.

The CPU 1131 uses the RAM 1132 as a work memory to execute programs stored in the ROM 1133 and the secondary storage device 1134, and comprehensively controls each component of the foreground generation server 113 via the system bus. As a result, the module shown in FIG. 1 is implemented, and the process shown in FIG. 2(a) is executed.

The secondary storage device 1134 is a device that stores various data handled by the foreground generation server 113, and uses, for example, an HDD, an optical disk drive, a flash memory, or the like. The CPU 1131 writes data to the secondary storage device 1134 and reads data stored in the secondary storage device 1134 via the system bus. The input/output interface 1135 transmits and receives data between the foreground generation server 113 and external devices. The above is the content of the system configuration in this embodiment.

For example, part of the processing of the foreground generation server 113 may be performed by an FPGA, and another part of the processing may be performed by software processing using a CPU. Further, each component of the foreground generation server 113 shown in FIG. 14 may be composed of a single electronic circuit, or may be composed of a plurality of electronic circuits. For example, the foreground generation server 113 may include multiple electronic circuits that operate as the CPU 1131 . The processing speed of the foreground generation server 113 can be improved by having these multiple electronic circuits perform processing as the CPU 1131 in parallel.

Other devices in the image processing system 100 are the same. Also, in the above-described embodiment, the example in which the image processing system 100 is installed in facilities such as stadiums and concert halls has been mainly described. Other examples of facilities include, for example, amusement parks, parks, racetracks, bicycle racetracks, casinos, swimming pools, skating rinks, ski resorts, and live music venues. Events held at various facilities may be held indoors or outdoors. In addition, the facility in this embodiment also includes a facility constructed temporarily (for a limited period of time).

[Other embodiments]
Also, the foreground generation server 113, the background generation system 120, and the rendering system 130 may physically have a plurality of servers inside, and may be configured to be logically one unit.

Further, in the processing of the above-described embodiments, the system or device may be provided with a storage medium recording program codes of software that embodies each function. Then, the computer (or CPU or MPU) of the system or device may read and execute the program code stored in the storage medium to realize the functions of the above-described embodiments. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Any storage medium may be used as a storage medium for supplying such program code, for example, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, or the like may be used.

Moreover, the functions of the above-described embodiments are not only realized by executing the program code read by the computer. Based on the instructions of the program code, an OS (operating system) or the like running on the computer may perform part or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Furthermore, the program code read from the storage medium may be written to a memory included in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU provided in the function expansion board or function expansion unit may perform part or all of the actual processing, and the functions of the above-described embodiments may be realized by the processing. .

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100... Image processing systems 110A, 110B... Foreground generation system 120... Background generation system

Claims (14)

An image processing device that generates a virtual viewpoint image,
a first foreground generation means for generating a foreground image based on an image captured by a first camera group composed of one or more cameras installed facing a first position;
Second foreground generation means for generating a foreground image based on an image captured by a second camera group composed of one or more cameras installed toward a second position different from the first position and,
Background generating means for generating a background image of the virtual viewpoint image based on a texture image for a background model generated based on the images captured by the first and second camera groups, and the background model. and
a third camera group composed of one or more cameras installed toward a third position different from the first and second positions;
has
The background generating means generates the texture image for the background model based on an image captured by the third camera group in addition to the images captured by the first and second camera groups. generating said background image further based on
An image processing apparatus characterized by : The background generation means generates the texture image for the background model based on the images captured by the first , second , and third camera groups, and synthesizes the generated texture image with the background model. The image processing apparatus according to claim 1, wherein the background image of the virtual viewpoint image is generated by: selecting means for selecting one of the first and second foreground generating means;
further comprising image generation means for generating the virtual viewpoint image based on the foreground image generated by the foreground generation means selected by the selection means and the background image generated by the background generation means; 3. The image processing apparatus according to claim 1, wherein The image generation means generates the virtual viewpoint image by synthesizing the foreground image generated by the foreground generation means selected by the selection means and the background image generated by the background generation means. 4. The image processing apparatus according to claim 3, wherein 5. The image processing according to claim 3, wherein said selection means selects either one of said first and said second foreground generation means according to the position and direction of a virtual viewpoint. Device. The background generation means converts the texture image generated based on the image of the camera group corresponding to the foreground generation means selected by the selection means to the camera corresponding to the foreground generation means not selected by the selection means. 6. The image processing apparatus according to any one of claims 3 to 5, wherein the background model is combined with a higher weight than the texture image generated based on the images of the group. The image of the first camera group for the first foreground generation means, the image of the second camera group for the second foreground generation means, the first and the second camera group for the background generation means, and 7. The image processing apparatus according to any one of claims 1 to 6, further comprising sorting means for sorting the images of the two camera groups. the camera belonging to the first or second camera group has separating means for separating and generating a foreground image and a background image from the photographed image;
6. The image processing apparatus according to claim 1, wherein said background generation means generates said texture image for said background model based on said background image generated by said separation means. . The background generating means may reduce the texture image generated based on the image of the third camera group to the texture image generated based on the image of the first and second camera groups. 9. The image processing apparatus according to any one of claims 1 to 8 , wherein the image is synthesized with the background model with high weight. 10. The camera according to any one of claims 1 to 9 , wherein the third camera group captures an image with a wider angle of view than the images captured by the first and second camera groups. The described image processing device. An image processing method for generating a virtual viewpoint image,
a first foreground generation step of generating a foreground image based on an image captured by a first camera group composed of one or more cameras installed toward a first position;
a second foreground generation step of generating a foreground image based on an image captured by a second camera group composed of one or more cameras installed toward a second position different from the first position; and,
A background generation step of generating a background image of the virtual viewpoint image based on a texture image for a background model generated based on the images captured by the first and second camera groups, and the background model. and
In the background generation step, in addition to the images captured by the first and second camera groups, one or more cameras installed toward a third position different from the first and second positions generating the background image further based on the texture image for the background model generated based on images taken by a third camera group composed of cameras;
An image processing method characterized by : In the background generation step, the texture image for the background model is generated based on the images captured by the first , second , and third camera groups, and the generated texture image is combined with the background model. 12. The image processing method according to claim 11 , wherein the background image of the virtual viewpoint image is generated by: a selection step of selecting one of the first and second foreground generation steps;
further comprising an image generation step of generating the virtual viewpoint image based on the foreground image generated by the foreground generation step selected in the selection step and the background image generated by the background generation step; 13. An image processing method according to claim 12 , characterized in that. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 10 .

Images