JP2019106145A - Generation device, generation method and program of three-dimensional model - Google Patents

Abstract

To prevent a defect from arising in a three-dimensional model generated even when a structure or the like that may shield a portion of a target object exists in an imaging scene.SOLUTION: A generation device of a three-dimensional model comprises: acquisition means that acquires a first mask image indicating a region of a structure being a stationary object in each image imaged by a plurality of viewpoints and a second mask image indicating a region of a foreground being an object of a moving object in each image imaged by the plurality of viewpoints; synthesis means that synthesizes the first mask image and the second mask image acquired to generate a third mask image integrating the region of the structure and the region of the foreground in the image imaged by the plurality of viewpoints; and generation means that generates the three-dimensional model including the structure and the foreground by means of a view volume crossing method using the third mask image.SELECTED DRAWING: Figure 4

Description

  The present invention relates to the generation of three-dimensional models of objects in an image.

  Conventionally, a method called "visual volume intersection method (Visual Hull)" is known as a method of estimating a three-dimensional shape of an object using a plurality of viewpoint images synchronously shot from different viewpoints by a plurality of cameras. (Patent Document 1). (A)-(c) of FIG. 1 is a figure which shows the basic principle of the visual volume intersection method. From an image obtained by photographing an object, a mask image representing a two-dimensional silhouette of the object is obtained on the imaging surface (FIG. 1A). Then, a cone that spreads in a three-dimensional space is considered so as to pass each point on the outline of the mask image from the projection center of the camera (FIG. 1 (b)). This cone is called the "view volume" of the object by the corresponding camera. Furthermore, the three-dimensional shape (three-dimensional model) of the object is determined by finding the intersection of a plurality of viewing volumes, that is, the intersection of the viewing volumes (FIG. 1 (c)). As described above, in shape estimation by the view volume intersection method, sampling points in a space in which an object may exist are projected onto a mask image, and whether a point projected in common by a plurality of viewpoints is included in the mask image By examining, the three-dimensional shape of the object is estimated.

JP, 2014-10805, A

  In the visual volume intersection method described above, the mask image needs to correctly represent the silhouette of the target object, and if the silhouette on the mask image is incorrect, the generated three-dimensional shape will also be incorrect. I will. For example, when a part of a person who is a target object is interrupted by a stationary object such as a structure existing in front of the person and a part of a person's silhouette indicated by a mask image is missing, a three-dimensional image is generated There will be a defect in the model. In addition, when not using a mask image in which a part of the silhouette is missing, the shape accuracy of the obtained three-dimensional model is degraded. In particular, when the portion blocked by the structure is relatively small, even if it is a mask image that is partially covered by the silhouette, it is possible to obtain a three-dimensional model with high shape accuracy, so it is used as much as possible. It is desirable to do.

  The present invention has been made in view of the above problems, and an object of the present invention is to generate a three-dimensional object even if a structure or the like that obstructs a part of a target object is present in a shooting scene. The goal is to ensure that no defects occur in the model.

  A three-dimensional model generation apparatus according to the present invention includes a first mask image indicating an area of a structure which is a stationary object in each image captured at a plurality of viewpoints, and each image captured at the plurality of viewpoints Acquisition means for acquiring a second mask image indicating a foreground area that is an object of a moving object in the image, and an image photographed from the plurality of viewpoints by combining the acquired first mask image and the acquired second mask image Combining the structure and the foreground by combining means for generating a third mask image in which the area of the structure and the foreground area are integrated and the third mask image using the third mask image And generating means for generating a three-dimensional model.

  According to the present invention, it is possible to generate a high-quality three-dimensional model free from defects or reduced even if a structure or the like that partially blocks a target object is present in a shooting scene. Become.

(A)-(c) is a figure which shows the basic principle of the visual volume intersection method (A) is a block diagram showing the configuration of a virtual viewpoint image generation system, (b) is a diagram showing an arrangement example of each camera constituting a camera array Functional block diagram showing the internal configuration of the three-dimensional model generation apparatus Flowchart showing a flow of three-dimensional model formation processing according to the first embodiment (A)-(h) is a figure which shows an example of the image image | photographed with each camera (A)-(h) is a figure which shows an example of a structure mask (A)-(h) is a figure which shows an example of a foreground mask (A)-(h) is a figure which shows an example of an integrated mask Diagram showing an example of an integrated 3D model generated based on an integrated mask A diagram showing an example of a three-dimensional model generated using only a foreground mask according to the conventional method Flowchart showing a flow of three-dimensional model formation processing according to the second embodiment (A) shows an integrated three-dimensional model generated based on an integrated mask, (b) shows a three-dimensional model of a structure generated based only on a structure mask, and (c) shows an integrated 3 of (a) A diagram showing a foreground-only three-dimensional model obtained by the difference between a two-dimensional model and the three-dimensional model of the structure of (b)

  Hereinafter, the present invention will be described in detail according to an embodiment with reference to the attached drawings. In addition, the structure shown in the following embodiment is only an example, and this invention is not limited to the illustrated structure.

Embodiment 1

  In the present embodiment, a three-dimensional model with no or reduced defects in the foreground using a mask image that also includes a two-dimensional silhouette of a structure that blocks at least a part of the foreground in addition to the two-dimensional silhouette of the foreground in a shooting scene The aspect which produces | generates is demonstrated. In this aspect, a three-dimensional model including a structure that partially blocks the foreground is generated. In the present specification, “foreground” refers to imaging that can be viewed from a virtual viewpoint that moves (when its absolute position may change) when imaging is performed from the same angle in time series. It refers to a dynamic object (moving object) present in an image. In addition, “structure” refers to shooting that may block the foreground if there is no movement (the absolute position does not change, ie, it is stationary) when shooting from the same angle in chronological order Refers to a static object present in the image.

  In the following description, when generating a virtual viewpoint image with a soccer game as a shooting scene, part of the foreground (dynamic object) such as a player or a ball is blocked by a structure (static object) such as a soccer goal. The case is assumed. The virtual viewpoint image is an image generated when the end user and / or an appointed operator freely manipulates the position and orientation of the virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image. In addition, the virtual viewpoint image to be generated and the multiple viewpoint images that are the origin of the virtual viewpoint image may be a moving image or a still image. In each embodiment described below, the case where a three-dimensional model for generating a virtual viewpoint image of a moving image is generated using a plurality of viewpoint images of the moving image will be described as an example.

  In the present embodiment, soccer is used as a shooting scene, and a soccer goal fixedly installed is described below as a structure, but the present invention is not limited to this. For example, the corner flag may be further treated as a structure, and when an indoor studio or the like is used as a shooting scene, furniture and props may be treated as a structure. That is, it may be a stationary object in which the stationary state or the near stationary state continues.

(System configuration)
FIG. 2A is a block diagram showing an example of the configuration of a virtual viewpoint image generation system including a three-dimensional model generation device according to the present embodiment. The virtual viewpoint image generation system 100 includes a camera array 110 including a plurality of cameras, a control device 120, a foreground separation device 130, a three-dimensional model generation device 140, and a rendering device 150. The control device 120, the foreground separation device 130, the three-dimensional model generation device 140, and the rendering device 150 generally include a CPU that performs arithmetic processing, a memory that stores results of arithmetic processing, programs, and the like. Realized by

  FIG. 2B is a view showing the arrangement of all eight cameras 211 to 218 constituting the camera array 110 in an overhead view when the field 200 is viewed from directly above. Each of the cameras 211 to 218 is installed to surround the field 200 at a certain height from the ground, and one of the goals is photographed from various angles to acquire multi-viewpoint image data having different viewpoints. On a grass field 200, a soccer court 201 is drawn (actually by a white line), and a soccer goal 202 is placed on the left side thereof. In addition, a cross mark 203 in front of the soccer goal 202 indicates a common line of sight direction (gaze point) of the cameras 211 to 218, and a circle 204 indicated by a broken line is an area where the cameras 211 to 218 can be photographed respectively Is shown. In this embodiment, a coordinate system in which the longitudinal direction is the x axis, the short direction is the y axis, and the height direction is the z axis, with one corner of the field 200 as the origin. Data of multiple source images obtained by each camera of the camera array 110 is sent to the control device 120 and the foreground separation device 130. In FIG. 2A, the cameras 211 to 218, and the control device 120 and the foreground separation device 130 are connected in a star topology, but may be a ring or bus topology by daisy chain connection. . Also, although FIG. 2 shows an example of eight cameras, the number of cameras may be less than eight or more than eight.

  The controller 120 generates a camera parameter and a structure mask, and supplies the camera parameter and the structure mask to the three-dimensional model generator 140. The camera parameters include external parameters representing the position and orientation (line of sight direction) of each camera, and internal parameters representing the focal length and angle of view (shooting area) of the lens of each camera, and are obtained by calibration. Calibration is a process of obtaining a correspondence between a point in a three-dimensional world coordinate system acquired using a plurality of images obtained by photographing a specific pattern such as a checkerboard and a corresponding two-dimensional point. The structure mask is a mask image showing a two-dimensional silhouette of a structure present in each captured image acquired by each of the cameras 211 to 218. The mask image is a reference image for specifying where the part to be extracted in the photographed image is, and is a binary image represented by 0 and 1. In the present embodiment, the soccer goal 202 is treated as a structure, and a silhouette image indicating an area (two-dimensional silhouette) of the soccer goal 202 in an image captured by each camera at a predetermined angle from a predetermined position is a structure mask. In addition, what is necessary is just to use what was image | photographed at the timing in which the player etc. who become a foreground, etc. do not exist, such as before and behind a game and in a half time, as the picked-up image used as a structure mask. However, for example, due to the influence of sunshine fluctuations outdoors, there may be cases where images taken in advance and after are inappropriate. In such a case, for example, a predetermined number of frames (for example, consecutive 10-second frames) of a moving image in which a player or the like appears may be used to delete the player or the like therefrom. In this case, a structure mask can be obtained based on an image adopting the median value of each pixel value in each frame.

  The foreground separation device 130 performs processing to discriminate a foreground area corresponding to a player or a ball on the field 200 and a background area other than that for each of the photographed images of a plurality of viewpoints to be input. For the determination of the foreground area, a background image prepared in advance (which may be the same as the photographed image which is the source of the structure mask) is used. Specifically, for each captured image, the difference with the background image is obtained, and the region corresponding to the difference is specified as the foreground region. Thereby, a foreground mask indicating the foreground area for each photographed image is generated. In the present embodiment, a binary image in which the pixels belonging to the foreground area representing the player or the ball in the photographed image are represented by “0” and the pixels belonging to the other background areas by “1” is generated as the foreground mask. It will be done.

  The three-dimensional model generation device 140 generates a three-dimensional model of the object based on the camera parameters and the multi-viewpoint image. Details of the three-dimensional model generation device 140 will be described later. The data of the generated three-dimensional model is output to the rendering device 150.

  The rendering device 150 is based on the three-dimensional model received from the three-dimensional model generator 140, the camera parameters received from the control device 120, the foreground image received from the foreground separation device 130, and the background image prepared in advance. Generate Specifically, the positional relationship between the foreground image and the three-dimensional model is obtained from the camera parameters, the foreground image corresponding to the three-dimensional model is mapped, and a virtual viewpoint image is generated when the object of interest is viewed from an arbitrary angle. Be done. Thus, for example, it is possible to obtain a virtual viewpoint image of a decisive scene before the goal that the player has scored.

  The configuration of the virtual viewpoint image generation system illustrated in FIG. 2 is an example and is not limited thereto. For example, one computer may have the functions of a plurality of devices (for example, the foreground separation device 130 and the three-dimensional model generation device 140). Alternatively, the module of each camera may be provided with the function of the foreground separation device 130, and the data of the photographed image and the foreground mask may be supplied from each camera.

(3D model generator)
FIG. 3 is a functional block diagram showing an internal configuration of the three-dimensional model generation device 140 according to the present embodiment. The three-dimensional model generation device 140 includes a data receiving unit 310, a structure mask storage unit 320, a mask combining unit 330, a coordinate conversion unit 340, a three-dimensional model forming unit 350, and a data output unit 360. Each part will be described in detail below.

  The data reception unit 310 receives, from the control device 120, the camera parameters of the respective cameras constituting the camera array 110 and the structure mask representing the two-dimensional silhouette of the structure present in the photographed scene. Also, from the foreground separation device 130, captured images (multi-viewpoint images) obtained by each camera of the camera array 110 and foreground mask data representing a two-dimensional silhouette of the foreground present in each captured image are received. Of the received data, the structure mask is for the structure mask storage unit 320, the foreground mask is for the mask combination unit 330, the multi-start point image is for the coordinate conversion unit 340, and the camera parameters are the coordinate conversion unit 340 and the three-dimensional model formation unit Passed to 350, respectively.

  The structure mask storage unit 320 stores and holds the structure mask in a RAM or the like, and supplies the structure mask to the mask combining unit 330 as necessary.

  The mask composition unit 330 reads out the structure mask from the structure mask storage unit 320, combines it with the foreground mask received from the data reception unit 310, and integrates both into one mask image (hereinafter referred to as “integrated mask To generate). The generated integrated mask is sent to the three-dimensional model formation unit 350.

  The coordinate conversion unit 340 converts the multi-viewpoint image received from the data reception unit 310 from the camera coordinate system to the world coordinate system based on the camera parameters. By this coordinate conversion, each photographed image from different viewpoints is converted into information indicating which region in the three-dimensional space is shown.

  The three-dimensional model formation unit 350 generates a three-dimensional model of an object including a structure in a photographed scene by the view volume intersection method using the multi-viewpoint image converted to the world coordinate system and the integrated mask corresponding to each camera. Do. The data of the generated three-dimensional model of the object is output to the rendering device 150 via the data output unit 360.

(Formation process of 3D model)
FIG. 4 is a flowchart showing a flow of three-dimensional model formation processing according to the present embodiment. This series of processing is realized by the CPU included in the three-dimensional model generation device 140 developing a predetermined program stored in a storage medium such as a ROM or an HDD in the RAM and executing the program. Hereinafter, description will be made along the flow of FIG.

  First, in step 401, the data receiving unit 310 represents a structure mask representing a two-dimensional silhouette of a structure (here, soccer goal 202) as viewed from each of the cameras 211 to 218, and a camera parameter of each camera. It receives from the control device 120. FIGS. 5A to 5H show images captured by the cameras 211 to 222 constituting the camera array 110, respectively. Now, one player (goal keeper) exists on the soccer court 201 in front of the soccer goal 202. And in each image pick-up picture of Drawing 5 (a), (b), (h), since soccer goal 202 is located between a camera and a player, a part of players may be covered by soccer goal 202. There is. From each photographed image in FIGS. 5A to 5H, a structure mask is obtained in which the area of the soccer goal 202 is represented by 1 (white) and the other area is expressed by 0 (black). Will be FIGS. 6A to 6H show structure masks corresponding to the photographed images of FIGS. 5A to 5H.

  Next, in step 402, a plurality of starting points from which the data receiving unit 310 is a foreground mask indicating a two-dimensional silhouette of the foreground (here, a player or a ball) in the image captured by each of the cameras 211 to 222 The image is received from the foreground separation device 130 together with the image. FIGS. 7 (a) to 7 (h) respectively show foreground masks corresponding to the photographed images of FIGS. 5 (a) to 5 (h). The foreground separation device 130 extracts a region having a temporal change between images captured from the same angle as the foreground, so in each of FIGS. 7 (a), (b) and (h), the soccer goal 202 Some areas of the player hidden in the are not extracted as foreground areas. The received foreground mask data is sent to the mask combining unit 330.

  Next, in step 403, the mask combining unit 310 reads out the data of the structure mask from the structure mask storage unit 320, and combines the read structure mask and the foreground mask received from the data receiving unit 310. Run. This combination is an arithmetic processing for obtaining a logical sum (OR) for each pixel of the foreground mask and the structure mask represented by binary (black and white). FIGS. 8 (a) to 8 (h) show the structure masks shown in FIGS. 6 (a) to 6 (h) and the foreground masks shown in FIGS. 7 (a) to 7 (h) respectively. The obtained integrated mask is shown. In the finished integrated mask, no loss is seen in the silhouette of the player.

  Then, at step 404, the three-dimensional model formation unit 350 generates a three-dimensional model using the visual volume intersection method based on the integrated mask obtained at step 403. As a result, a model (hereinafter referred to as “integrated three-dimensional model”) representing the three-dimensional shape of the foreground and the structure present in the common imaging region between the plurality of images captured from different viewpoints is generated. In the case of this embodiment, an integrated three-dimensional model including the soccer goal 202 is generated in addition to the player and the ball. Specifically, the generation of the integrated three-dimensional model is performed according to the following procedure. First, volume data is prepared by filling a three-dimensional space on the field 200 with a cube (voxel) having a certain size. The values of the voxels constituting the volume data are expressed by 0 and 1, respectively, “1” indicates a shape area, and “0” indicates a non-shape area. Next, the three-dimensional coordinates of the voxel are converted from the world coordinate system to the camera coordinate system using the camera parameters (such as the installation position and the line of sight direction) of each of the cameras 211 to 218. Then, when the structure indicated by the integrated mask and the foreground are in the camera coordinate system, a model is generated which represents the three-dimensional shape of the structure and the foreground by voxels. The three-dimensional shape may be expressed not by voxels themselves but by a set of points (point group) indicating the centers of the voxels. FIG. 9 shows an integrated three-dimensional model generated based on the integrated mask shown in FIG. 8, in which reference numeral 901 denotes the three-dimensional shape of the player who is the foreground, and reference numeral 902 denotes the soccer goal 202 which is a structure. It corresponds to a dimensional shape. As described above, since the integrated mask has no loss in the foreground player silhouette, no loss occurs in the completed integrated three-dimensional model. FIG. 10 shows a three-dimensional model generated using only the foreground mask according to the conventional method. As described above, in the foreground masks shown in (a), (b) and (h) of FIG. 7, a part of the player is not represented as a foreground area, so the part is missing in the generated three-dimensional model Resulting in. In the method of the present embodiment, it is possible to avoid the occurrence of a defect in a part of the three-dimensional model of the foreground by using a mask image obtained by combining the foreground mask and the structure mask.

  The above is the contents of the three-dimensional model formation processing according to the present embodiment. In the case of generating a virtual viewpoint image of a moving image, the processing of each step described above is repeatedly performed in frame units to generate a three-dimensional model for each frame. However, it is sufficient to receive and save the structure mask (step 401) only immediately after the start of the flow, and the second and subsequent frames can be omitted. Furthermore, in the case where shooting is performed by changing the date and time at the same shooting location, reception and storage of the structure mask are performed only for the first time, held in the RAM, etc., and those held from the next time on are used. You may

  As described above, according to the present embodiment, even if there is a structure that hides the foreground object, it is possible to generate a highly accurate three-dimensional model with no or reduced loss in the foreground.

Embodiment 2

  In the first embodiment, a three-dimensional model of a defect-free or reduced foreground is generated so as to include structures present in a photographed scene. Next, an aspect of generating a three-dimensional model with no defect or reduced or reduced foreground only will be described as a second embodiment. The contents common to the first embodiment, such as the system configuration, will be omitted or simplified, and in the following, differences will be mainly described.

  The configuration of the three-dimensional model generation apparatus 140 of the present embodiment is also basically the same as that of the first embodiment (see FIG. 3), but differs in the following points.

  First, readout of the structure mask from the structure mask storage unit 320 is performed not only by the mask combining unit 330 but also by the three-dimensional model generation unit 350. The dashed arrow in FIG. 3 represents this. Then, in addition to the generation of the integrated three-dimensional model of the foreground + the structure using the integrated mask, the three-dimensional model generation unit 350 also generates the three-dimensional model of only the structure using the structure mask. Then, the difference between the integrated three-dimensional model generated based on the integrated mask and the three-dimensional model of the structure generated based on the structure mask is calculated to obtain the three-dimensional of the foreground without defects or reduced. Extract the model

(Formation process of 3D model)
FIG. 11 is a flowchart showing a flow of three-dimensional model formation processing according to the present embodiment. This series of processing is realized by the CPU included in the three-dimensional model generation device 140 developing a predetermined program stored in a storage medium such as a ROM or an HDD in the RAM and executing the program. Hereinafter, description will be made along the flow of FIG.

  Steps 1101 to 1104 correspond to steps 401 to 404 in the flow of FIG. 4 of the first embodiment, respectively, and the description will be omitted because there is no difference.

  In the subsequent step 1105, the three-dimensional model formation unit 350 reads the structure mask from the structure mask storage unit 320, and generates a three-dimensional model of the structure by the visual volume intersection method.

  Next, at step 1106, the three-dimensional model formation unit 350 obtains the difference between the combined three-dimensional model of the foreground + structure generated at step 1104 and the three-dimensional model of the structure generated at step 1105 Extract a 3D model. Here, the three-dimensional model of the structure may be expanded by, for example, about 10% in three-dimensional space, and then the difference with the integrated three-dimensional model may be obtained. This makes it possible to reliably remove the part corresponding to the structure from the integrated three-dimensional model. At this time, only a part of the three-dimensional model of the structure may be expanded. For example, in the case of the soccer goal 202, since there is a high possibility that a player is present in the soccer court 201, the player is not allowed to inflate on the court 201 side, and is inflated only on the opposite side to the court 201. The portion to be inflated may be determined according to Furthermore, the expansion rate (expansion rate) may be changed depending on how far the foreground object such as the player or the ball is from the structure. For example, when the foreground object is at a position far from the structure, increasing the expansion rate ensures that the three-dimensional model of the structure is removed. Also, by decreasing the expansion rate as the foreground object is closer to the structure, the foreground three-dimensional model portion is prevented from being erroneously removed. The expansion rate at this time may be changed linearly according to the distance from the foreground, or may be determined stepwise by providing one or more reference distances.

  FIG. 12A shows an integrated three-dimensional model generated based on the integrated mask, as in FIG. 9 described above. FIG. 12 (b) shows a three-dimensional model of a structure generated based only on the structure mask. And FIG.12 (c) has shown the three-dimensional model of only a foreground obtained by the difference of the integrated three-dimensional model of Fig.12 (a), and the three-dimensional model of the structure of FIG.12 (b). .

  The above is the contents of the process of forming a three-dimensional model according to the present embodiment. In the case of generating a virtual viewpoint image of a moving image, the processing of each step described above is repeatedly performed in frame units to generate a three-dimensional model for each frame. However, the reception and storage of the structure mask (step 1101) and the generation of the three-dimensional model of the structure (step 1105) may be performed only immediately after the start of the flow, and may be omitted for the second and subsequent frames. Furthermore, when changing the date and time at the same shooting location and shooting, the reception and storage of the structure mask and the three-dimensional model generation of the structure are performed only for the first time and held in the RAM etc. You may use what you hold.

(Modification)
In the present embodiment, the three-dimensional model of only the foreground is generated by subtracting the three-dimensional model of the structure from the integrated three-dimensional model of the foreground + the structure, but the present invention is not limited thereto. For example, it counts which mask image is included in each voxel (or each predetermined area) which composes an integrated 3D model of foreground + structure, and deletes a portion having a count value equal to or less than a threshold from the integrated 3D model The foreground only three-dimensional model may be obtained by The threshold value in this case is set to an arbitrary value smaller than the total number of cameras in consideration of the installation position of each camera, the line of sight direction, and the like. In the case of the present embodiment in which the number of cameras is eight and the camera arrangement is as shown in FIG. 2A, only the soccer goal can be deleted by setting, for example, "2" as the threshold value.

  As described above, according to the present embodiment, even if there is a structure that hides the foreground object, it is possible to generate a highly accurate three-dimensional model of the foreground only, which does not include the structure.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

140 Three-Dimensional Model Generator 310 Data Receiver 330 Mask Synthesizer 350 Three-Dimensional Model Generator

  The generation apparatus according to the present invention comprises: a first acquisition unit for acquiring first area information indicating an area of an object in a plurality of images obtained by photographing from a plurality of photographing directions; and at least one of the plurality of photographing directions. A second acquisition unit that acquires second area information indicating an area of a structure that may obstruct the object at the time of imaging from one imaging direction; and a first area that indicates the area of the object acquired by the first acquisition unit And generating means for generating three-dimensional shape data corresponding to the object based on both the information and the second area information indicating the area of the structure acquired by the second acquiring means.

FIG. 2B is a view showing the arrangement of all eight cameras 211 to 218 constituting the camera array 110 in an overhead view when the field 200 is viewed from directly above. Each of the cameras 211 to 218 is installed to surround the field 200 at a certain height from the ground, and one of the goals is photographed from various angles to acquire multi-viewpoint image data having different viewpoints. On a grass field 200, a soccer court 201 is drawn (actually by a white line), and a soccer goal 202 is placed on the left side thereof. In addition, a cross mark 203 in front of the soccer goal 202 indicates a common line of sight direction (gaze point) of the cameras 211 to 218, and a circle 204 indicated by a broken line is an area where the cameras 211 to 218 can be photographed centering on the gaze point 203. Is shown. In this embodiment, a coordinate system in which the longitudinal direction is the x axis, the short direction is the y axis, and the height direction is the z axis, with one corner of the field 200 as the origin. Data of multi- viewpoint images obtained by each camera of the camera array 110 is sent to the control device 120 and the foreground separation device 130. In FIG. 2A, the cameras 211 to 218, and the control device 120 and the foreground separation device 130 are connected in a star topology, but may be a ring or bus topology by daisy chain connection. . Also, although FIG. 2 shows an example of eight cameras, the number of cameras may be less than eight or more than eight.

The data reception unit 310 receives, from the control device 120, the camera parameters of the respective cameras constituting the camera array 110 and the structure mask representing the two-dimensional silhouette of the structure present in the photographed scene. Also, from the foreground separation device 130, captured images (multi-viewpoint images) obtained by each camera of the camera array 110 and foreground mask data representing a two-dimensional silhouette of the foreground present in each captured image are received. Among the received data, the structure mask is for the structure mask storage unit 320, the foreground mask is for the mask combination unit 330, the multi- viewpoint image is for the coordinate conversion unit 340, and the camera parameters are the coordinate conversion unit 340 and the three-dimensional model formation unit Passed to 350, respectively.

Next, in step 402, the data receiving unit 310 is a plurality of viewpoints from which a foreground mask indicating a two-dimensional silhouette of the foreground (here, a player or a ball) in the image captured by each of the cameras 211 to 218 is derived . It is received from the foreground separation device 130 together with the image. FIGS. 7 (a) to 7 (h) respectively show foreground masks corresponding to the photographed images of FIGS. 5 (a) to 5 (h). The foreground separation device 130 extracts a region having a temporal change between images captured from the same angle as the foreground, so in each of FIGS. 7 (a), (b) and (h), the soccer goal 202 Some areas of the player hidden in the are not extracted as foreground areas. The received foreground mask data is sent to the mask combining unit 330.

Claims (10)

A first mask image showing a region of a structure which is a stationary object in each image photographed at a plurality of viewpoints, and a foreground region which is an object of a moving object in each image photographed at each of the plurality of viewpoints Acquisition means for acquiring a second mask image;
The acquired first mask image and the second mask image are combined to generate a third mask image in which an area of the structure and an area of the foreground in the images captured at the plurality of viewpoints are integrated. Synthesis means,
Generation means for generating a three-dimensional model including the structure and the foreground by a view volume intersection method using the third mask image;
An apparatus for generating a three-dimensional model, comprising:   The generation apparatus according to claim 1, wherein the combining unit performs the combining by obtaining a logical sum of the first mask image and the second mask image. The foreground area is an area in which a dynamic object whose position can change in each image when the photographing is performed in time series from the same angle is reflected,
The area of the structure is an area where there is a static object which may block at least a part of the foreground, whose position does not change in each image when the photographing is performed in time series from the same angle The generating device according to claim 1 or 2, characterized in that: The generation means is
A three-dimensional model of the structure is further generated by a view volume intersection method using the first mask image,
A three-dimensional model of the foreground is generated from a difference between the generated three-dimensional model of the structure and the integrated three-dimensional model.
The generator according to any one of claims 1 to 3, wherein   The said generation means calculates | requires the difference with the said integrated three-dimensional model using the three-dimensional model of the said structure which expanded the at least one part on three-dimensional space, The said three-dimensional model Generator of.   The generating device according to claim 5, wherein the generating means determines an expanding portion of a three-dimensional model of the structure in accordance with a region in the three-dimensional space in which the structure exists.   The generation apparatus according to claim 5, wherein the generation means determines an expansion ratio of a three-dimensional model of the structure in accordance with a distance between the structure and the foreground.   8. The generation apparatus according to claim 7, wherein the generation unit increases the expansion ratio of the three-dimensional model of the structure as the structure is farther from the foreground. Acquiring a first mask image indicating a region of a structure in each image captured at a plurality of viewpoints, and acquiring a second mask image indicating a region of a foreground in each image captured at the plurality of viewpoints;
The acquired first mask image and the second mask image are combined to generate a third mask image in which an area of the structure and an area of the foreground in the images captured at the plurality of viewpoints are integrated. A synthesis step,
Generating a three-dimensional model including the structure and the foreground by a view volume intersection method using the third mask image;
A method of generating a three-dimensional model, comprising:   A program for causing a computer to function as the generation device according to any one of claims 1 to 8.

Images