JP6403862B1 - Three-dimensional model generation apparatus, generation method, and program - Google Patents

Abstract

An object of the present invention is to prevent a defect from being generated in a generated three-dimensional model even if a structure or the like that blocks a part of a target object exists in a shooting scene. An apparatus for generating a three-dimensional model, a first mask image showing a region of a structure that is a stationary object in each image photographed from a plurality of viewpoints, and photographed from the plurality of viewpoints. Photographing from the plurality of viewpoints by synthesizing the obtained first mask image and the second mask image with an obtaining means for obtaining a second mask image indicating a foreground region which is a moving object in each image The structure and the foreground are generated by combining means for generating a third mask image in which the region of the structure and the foreground region in the obtained image are integrated, and a view volume intersection method using the third mask image. Generating means for generating a three-dimensional model including [Selection] Figure 4

Description

  The present invention relates to generation of a three-dimensional model of an object in an image.

  Conventionally, a technique called “visual volume crossing method (Visual Hull)” is known as a technique for estimating the three-dimensional shape of an object using a plurality of viewpoint images synchronously photographed from different viewpoints by a plurality of cameras. (Patent Document 1). (A)-(c) of FIG. 1 is a figure which shows the basic principle of a visual volume intersection method. From an image obtained by photographing an object, a mask image representing a two-dimensional silhouette of the object is obtained on the imaging surface (FIG. 1A). Then, consider a cone extending in the three-dimensional space so that each point on the contour of the mask image passes from the projection center of the camera (FIG. 1B). This cone is called the “viewing volume” of the object by the corresponding camera. Furthermore, the three-dimensional shape (three-dimensional model) of the object is obtained by obtaining the common region of the plurality of viewing volumes, that is, the intersection of the viewing volumes (FIG. 1C). As described above, in the shape estimation by the visual volume intersection method, sampling points in a space where an object may exist are projected onto the mask image, and whether the points projected in common at a plurality of viewpoints are included in the mask image. By verifying, the three-dimensional shape of the object is estimated.

JP 2014-10805 A

  In the visual volume intersection method described above, the mask image needs to correctly represent the silhouette of the target object, and if the silhouette on the mask image is incorrect, the generated three-dimensional shape is also incorrect. End up. For example, when a part of a person who is a target object is blocked by a stationary object such as a structure existing in front of the person and a part of the silhouette of the person indicated by the mask image is missing, the generated three-dimensional The model is deficient. If the mask image lacking a part of the silhouette is not used, the shape accuracy of the obtained three-dimensional model is lowered. In particular, if the part obstructed by the structure is relatively small, even if it is a mask image with a part of the silhouette, a 3D model with high shape accuracy can be obtained by using it as much as possible. It is desirable to do.

  The present invention has been made in view of the above problems, and its purpose is to generate a three-dimensional image even if a structure or the like that blocks a part of the target object exists in the shooting scene. It is to prevent defects from occurring in the model.

  The generation apparatus according to the present invention includes first acquisition means for acquiring first area information indicating an area of an object in a plurality of images obtained by shooting from a plurality of shooting directions, and at least one of the plurality of shooting directions. Second acquisition means for acquiring second area information indicating an area of a structure that may block the object during shooting from one shooting direction; and a first area indicating the area of the object acquired by the first acquisition means And generating means for generating three-dimensional shape data corresponding to the object based on both the information and the second area information indicating the area of the structure acquired by the second acquiring means.

  According to the present invention, it is possible to generate a high-quality three-dimensional model that is free from defects or reduced, even if a structure or the like that blocks a part of the target object exists in the shooting scene. Become.

(A)-(c) is a figure which shows the basic principle of a visual volume intersection method (A) is a block diagram showing a configuration of a virtual viewpoint image generation system, (b) is a diagram showing an arrangement example of each camera constituting a camera array Functional block diagram showing the internal configuration of the three-dimensional model generation device The flowchart which shows the flow of the three-dimensional model formation process based on Embodiment 1. FIG. (A)-(h) is a figure which shows an example of the image image | photographed with each camera. (A)-(h) is a figure which shows an example of a structure mask (A)-(h) is a figure which shows an example of a foreground mask (A)-(h) is a figure which shows an example of an integrated mask. The figure which shows an example of the integrated three-dimensional model produced | generated based on an integrated mask The figure which shows an example of the three-dimensional model produced | generated using only the foreground mask by the conventional method The flowchart which shows the flow of the three-dimensional model formation process based on Embodiment 2. FIG. (A) shows the integrated three-dimensional model generated based on the integrated mask, (b) shows the three-dimensional model of the structure generated based only on the structure mask, and (c) shows the integrated three-dimensional model of (a). The figure which shows the three-dimensional model of only the foreground obtained by the difference of a three-dimensional model of a three-dimensional model and the structure of (b)

  Hereinafter, the present invention will be described in detail according to embodiments with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

Embodiment 1

  In the present embodiment, a foreground is free from or reduced in 3D model using a mask image including a 2D silhouette of a foreground in a shooting scene and a 2D silhouette of a structure that blocks at least a part thereof. A mode of generating the will be described. In this aspect, a three-dimensional model including a structure that blocks a part of the foreground is generated. Note that in this specification, “foreground” refers to an image that can be viewed from a virtual viewpoint that has movement (its absolute position may change) when imaged from the same angle in time series. A dynamic object (moving object) existing in an image. In addition, “structure” means that there is no movement when shooting from the same angle in time series (the absolute position does not change, that is, it is stationary), which may block the foreground. A static object that exists in an image.

  In the following description, when a virtual viewpoint image is generated using a soccer game as a shooting scene, a part of the foreground (dynamic object) such as a player or a ball is blocked by a structure (static object) such as a soccer goal. This assumes a case where Note that a virtual viewpoint image is a video generated by an end user and / or a selected operator who freely manipulates the position and orientation of a virtual camera, and is also referred to as a free viewpoint image or an arbitrary viewpoint image. In addition, the generated virtual viewpoint image and the multiple viewpoint images that are the basis thereof may be a moving image or a still image. In each embodiment described below, a case where a three-dimensional model for generating a virtual viewpoint image of a moving image is generated using a plurality of viewpoint images of the moving image will be described as an example.

  In the present embodiment, soccer is taken as a shooting scene, and a soccer goal fixedly set is assumed to be a structure. However, the present invention is not limited to this. For example, the corner flag may be further handled as a structure, and furniture or props can be handled as a structure when an indoor studio or the like is used as a shooting scene. In other words, it may be a stationary object that continues to be stationary or nearly stationary.

(System configuration)
FIG. 2A is a block diagram illustrating an example of a configuration of a virtual viewpoint image generation system including a three-dimensional model generation apparatus according to the present embodiment. The virtual viewpoint image generation system 100 includes a camera array 110 including a plurality of cameras, a control device 120, a foreground separation device 130, a three-dimensional model generation device 140, and a rendering device 150. The control device 120, the foreground separation device 130, the three-dimensional model generation device 140, and the rendering device 150 are each a general computer (information processing device) that includes a CPU that performs arithmetic processing, a memory that stores arithmetic processing results, programs, and the like. ).

FIG. 2B is a diagram showing an arrangement of all eight cameras 211 to 218 constituting the camera array 110 in an overhead view when the field 200 is viewed from directly above. Each of the cameras 211 to 218 is installed so as to surround the field 200 at a certain height from the ground, and captures a plurality of viewpoint image data with different viewpoints by photographing one goal front from various angles. On the grass field 200, a soccer court 201 is drawn (actually with a white line), and a soccer goal 202 is placed on the left side. A cross mark 203 in front of the soccer goal 202 indicates a common gaze direction (gaze point) of the cameras 211 to 218, and a broken circle 204 is an area where the cameras 211 to 218 can shoot around the gaze point 203. Is shown. In the present embodiment, the coordinate system is represented by a coordinate system in which one corner of the field 200 is the origin, the longitudinal direction is the x axis, the lateral direction is the y axis, and the height direction is the z axis. Multi- viewpoint image data obtained by each camera of the camera array 110 is sent to the control device 120 and the foreground separation device 130. In FIG. 2A, the cameras 211 to 218, the control device 120, and the foreground separation device 130 are connected in a star type topology, but may be a ring type or bus type topology by daisy chain connection. . Moreover, although the example of eight cameras was shown in FIG. 2, the number of cameras may be less than eight or more than eight.

  The control device 120 generates camera parameters and a structure mask and supplies them to the three-dimensional model generation device 140. The camera parameters include external parameters that indicate the position and orientation (line-of-sight direction) of each camera, and internal parameters that indicate the focal length and angle of view (shooting area) of the lens provided in each camera, and are obtained by calibration. The calibration is a process for obtaining a correspondence relationship between a point in the three-dimensional world coordinate system acquired using a plurality of images obtained by capturing a specific pattern such as a checkerboard and a corresponding two-dimensional point. The structure mask is a mask image showing a two-dimensional silhouette of a structure existing in each captured image acquired by each camera 211 to 218. The mask image is a reference image that specifies where the portion to be extracted in the photographed image is, and is a binary image represented by 0 and 1. In the present embodiment, the soccer goal 202 is treated as a structure, and a silhouette image indicating a region (two-dimensional silhouette) of the soccer goal 202 in an image captured by each camera at a predetermined angle from a predetermined position is a structure mask. In addition, what is necessary is just to use the image | photographed image used as the origin of a structure mask image | photographed at the timing when the player etc. which become a foreground do not exist, such as before and after a game, and half time. However, there are cases where images taken before and after the event are inappropriate due to, for example, the influence of sunshine fluctuations outdoors. In such a case, for example, it may be obtained by using a predetermined number of frames (for example, frames for 10 consecutive seconds) in a moving image in which a player or the like is shown and erasing the player or the like from there. In this case, a structure mask can be obtained based on an image employing the median value of each pixel value in each frame.

  The foreground separation device 130 performs a process of discriminating a foreground area corresponding to a player or a ball on the field 200 and a background area other than that for each of the input images of the plurality of viewpoints. For the discrimination of the foreground region, a background image prepared in advance (which may be the same as the captured image that is the basis of the structure mask) is used. Specifically, a difference from the background image is obtained for each captured image, and an area corresponding to the difference is specified as a foreground area. Thus, a foreground mask indicating the foreground area for each captured image is generated. In the present embodiment, a binary image in which a pixel belonging to the foreground area representing the player or the ball in the captured image is represented by “0” and a pixel belonging to the other background area represented by “1” is generated as the foreground mask. Will be.

  The three-dimensional model generation device 140 generates a three-dimensional model of an object based on camera parameters and multiple viewpoint images. Details of the three-dimensional model generation apparatus 140 will be described later. The generated three-dimensional model data is output to the rendering device 150.

  The rendering device 150 generates a virtual viewpoint image based on the three-dimensional model received from the three-dimensional model generation device 140, the camera parameters received from the control device 120, the foreground image received from the foreground separation device 130, and a background image prepared in advance. Generate. Specifically, the positional relationship between the foreground image and the 3D model is obtained from the camera parameters, the foreground image corresponding to the 3D model is mapped, and a virtual viewpoint image is generated when the object of interest is viewed from an arbitrary angle. Is done. Thus, for example, a virtual viewpoint image of a definitive scene before the goal where the player has scored can be obtained.

  The configuration of the virtual viewpoint image generation system shown in FIG. 2 is an example and is not limited to this. For example, one computer may have the functions of a plurality of devices (for example, the foreground separation device 130 and the three-dimensional model generation device 140). Alternatively, each camera module may be provided with the function of the foreground separation device 130, and each camera may be configured to supply the captured image and its foreground mask data.

(3D model generator)
FIG. 3 is a functional block diagram showing the internal configuration of the three-dimensional model generation apparatus 140 according to this embodiment. The three-dimensional model generation apparatus 140 includes a data reception unit 310, a structure mask storage unit 320, a mask composition unit 330, a coordinate conversion unit 340, a three-dimensional model formation unit 350, and a data output unit 360. Hereinafter, each part will be described in detail.

The data receiving unit 310 receives from the control device 120 camera parameters of each camera constituting the camera array 110 and a structure mask representing a two-dimensional silhouette of the structure existing in the shooting scene. Further, the foreground mask data representing the captured images (multiple viewpoint images) obtained by the respective cameras of the camera array 110 and the two-dimensional silhouette of the foreground existing in the captured images is received from the foreground separation device 130. Of the received data, the structure mask is stored in the structure mask storage unit 320, the foreground mask is stored in the mask composition unit 330, the multi- viewpoint image is stored in the coordinate conversion unit 340, and the camera parameters are stored in the coordinate conversion unit 340 and the 3D model formation unit. 350, respectively.

  The structure mask storage unit 320 stores and holds the structure mask in a RAM or the like, and supplies the structure mask to the mask composition unit 330 as necessary.

  The mask composition unit 330 reads the structure mask from the structure mask storage unit 320, combines it with the foreground mask received from the data reception unit 310, and integrates the two into one mask image (hereinafter, “integrated mask”). "). The generated integrated mask is sent to the three-dimensional model forming unit 350.

  The coordinate conversion unit 340 converts the multiple viewpoint images received from the data reception unit 310 from the camera coordinate system to the world coordinate system based on the camera parameters. By this coordinate conversion, each captured image having a different viewpoint is converted into information indicating which region in the three-dimensional space is indicated.

  The three-dimensional model forming unit 350 generates a three-dimensional model of an object including a structure in a shooting scene by a view-volume intersection method using a plurality of viewpoint images converted to the world coordinate system and an integrated mask corresponding to each camera. To do. Data of the generated three-dimensional model of the object is output to the rendering device 150 via the data output unit 360.

(Three-dimensional model formation process)
FIG. 4 is a flowchart showing the flow of the three-dimensional model formation process according to this embodiment. This series of processing is realized by a CPU provided in the three-dimensional model generation apparatus 140 developing a predetermined program stored in a storage medium such as a ROM or HDD on the RAM and executing the program. Hereinafter, it demonstrates along the flow of FIG.

  First, in step 401, the data receiving unit 310 obtains a structure mask representing a two-dimensional silhouette of a structure (here, soccer goal 202) when viewed from each camera 211 to 218, and camera parameters of each camera. Received from the control device 120. 5A to 5H show images captured by the cameras 211 to 222 constituting the camera array 110, respectively. Now, one player (goal keeper) is present in front of the soccer goal 202 on the soccer court 201. 5A, 5B, and 5H, the soccer goal 202 is positioned between the camera and the player, so that a part of the player is hidden by the soccer goal 202. Yes. Each of the captured images of FIGS. 5 (a) to 5 (h) obtains a structure mask in which the soccer goal 202 area is expressed as a binary value of 1 (white) and the other areas are expressed as 0 (black). Will be. FIGS. 6A to 6H show structure masks corresponding to the captured images shown in FIGS. 5A to 5H.

Next, in step 402, the data receiving unit 310 uses a plurality of viewpoints based on a foreground mask indicating a two-dimensional silhouette of a foreground (here, a player or a ball) in an image captured by each of the cameras 211 to 218. It is received from the foreground separation device 130 together with the image. FIGS. 7A to 7H show foreground masks corresponding to the captured images of FIGS. 5A to 5H, respectively. Since the foreground separation device 130 extracts, as the foreground, a region that changes in time between images taken from the same angle, in each of FIGS. 7A, 7B, and 7H, the soccer goal 202 is displayed. Some areas of players hidden behind are not extracted as foreground areas. The received foreground mask data is sent to the mask composition unit 330.

  Next, in step 403, the mask composition unit 310 reads the structure mask data from the structure mask storage unit 320, and synthesizes the read structure mask and the foreground mask received from the data reception unit 310. Run. This synthesis is an arithmetic process for obtaining a logical sum (OR) for each pixel of the foreground mask and the structure mask expressed in binary (black and white). 8A to 8H are obtained by combining the structure masks shown in FIGS. 6A to 6H and the foreground masks shown in FIGS. 7A to 7H, respectively. The resulting integrated mask is shown. In the completed integrated mask, there are no defects in the silhouette of the players.

  In step 404, the three-dimensional model forming unit 350 generates a three-dimensional model using the visual volume intersection method based on the integrated mask obtained in step 403. Thereby, a model (hereinafter referred to as “integrated three-dimensional model”) representing the three-dimensional shape of the foreground and the structure existing in a common photographing region between a plurality of images photographed from different viewpoints is generated. In the case of the present embodiment, an integrated three-dimensional model including the soccer goal 202 in addition to the player and the ball is generated. Specifically, the integrated three-dimensional model is generated in the following procedure. First, volume data in which a three-dimensional space on the field 200 is filled with a cube (voxel) having a certain size is prepared. The values of voxels constituting the volume data are represented by 0 and 1, where “1” indicates a shape area and “0” indicates a non-shape area. Next, the three-dimensional coordinates of the voxel are converted from the world coordinate system to the camera coordinate system using the camera parameters (installation position, line-of-sight direction, etc.) of each camera 211 to 218. When the structure and foreground indicated by the integrated mask are in the camera coordinate system, a model representing the three-dimensional shape of the structure and foreground is generated by the voxel. Note that the three-dimensional shape may be expressed not by the voxel itself but by a set of points (point group) indicating the center of the voxel. FIG. 9 shows an integrated three-dimensional model generated based on the integrated mask shown in FIG. 8. Reference numeral 901 is the three-dimensional shape of the player as the foreground, and reference numeral 902 is the 3 of the soccer goal 202 as the structure. Corresponds to a dimensional shape. As described above, since there is no defect in the silhouette of the player as the foreground in the integrated mask, there is no defect in the completed integrated three-dimensional model. FIG. 10 shows a three-dimensional model generated using only the foreground mask according to the conventional method. As described above, in the foreground mask shown in FIGS. 7A, 7 </ b> B, and 7 </ b> H, a part of the player is not represented as the foreground region, and therefore the part is missing in the generated three-dimensional model. Resulting in. In the method of the present embodiment, it is possible to avoid the occurrence of a defect in a part of the three-dimensional model of the foreground by using a mask image obtained by combining the foreground mask and the structure mask.

  The above is the content of the three-dimensional model formation process according to the present embodiment. When generating a virtual viewpoint image of a moving image, the processing of each step described above is repeated for each frame to generate a three-dimensional model for each frame. However, the reception and storage of the structure mask (step 401) need only be performed immediately after the start of the flow, and can be omitted for the second and subsequent frames. Furthermore, when shooting at the same shooting location with different dates and times, the structure mask is received and saved only for the first time and stored in RAM, etc. May be.

  As described above, according to the present embodiment, it is possible to generate a highly accurate three-dimensional model that is free from or reduced in the foreground even if there is a structure that hides the object that is the foreground.

Embodiment 2

  In the first embodiment, a three-dimensional model of the foreground that is free of defects or reduced is generated in a form that includes structures present in the shooting scene. Next, a mode of generating a three-dimensional model with only the foreground without a defect or with a reduced structure will be described as a second embodiment. The contents common to the first embodiment such as the system configuration will not be described or simplified, and the differences will be mainly described below.

  The configuration of the three-dimensional model generation apparatus 140 of the present embodiment is basically the same as that of the first embodiment (see FIG. 3), but differs in the following points.

  First, the structure mask is read out from the structure mask storage unit 320 not only by the mask composition unit 330 but also by the three-dimensional model generation unit 350. The broken arrow in FIG. 3 represents this. In addition to the generation of the foreground + structure integrated 3D model using the integrated mask, the 3D model generation unit 350 also generates a 3D model of only the structure using the structure mask. Then, by obtaining a difference between the integrated three-dimensional model generated based on the integrated mask and the three-dimensional model of the structure generated based on the structure mask, there is no missing or reduced three-dimensional foreground only. Extract the model.

(Three-dimensional model formation process)
FIG. 11 is a flowchart showing the flow of the three-dimensional model formation process according to this embodiment. This series of processing is realized by a CPU provided in the three-dimensional model generation apparatus 140 developing a predetermined program stored in a storage medium such as a ROM or HDD on the RAM and executing the program. Hereinafter, it demonstrates along the flow of FIG.

  Steps 1101 to 1104 correspond to Steps 401 to 404 in the flow of FIG. 4 of the first embodiment, respectively, and are not different, so description thereof is omitted.

  In subsequent step 1105, the three-dimensional model forming unit 350 reads the structure mask from the structure unit mask storage unit 320 and generates a three-dimensional model of the structure by the visual volume intersection method.

  Next, in step 1106, the three-dimensional model forming unit 350 obtains a difference between the foreground + structure synthesized three-dimensional model generated in step 1104 and the three-dimensional model of the structure generated in step 1105, and calculates only the foreground. A three-dimensional model is extracted. Here, the difference from the integrated three-dimensional model may be obtained after expanding the three-dimensional model of the structure in the three-dimensional space by, for example, about 10%. Thereby, the part corresponding to a structure can be reliably removed from the integrated three-dimensional model. At this time, only a part of the three-dimensional model of the structure may be expanded. For example, in the case of the soccer goal 202, since there is a high possibility that a player exists in the soccer court 201, the area is not expanded on the side of the court 201 and only on the side opposite to the court 201. The portion to be expanded may be determined according to the above. Further, the rate of expansion (expansion rate) may be changed depending on how far the foreground object such as a player or ball is away from the structure. For example, when the foreground object is at a position far from the structure, the three-dimensional model of the structure is surely removed by increasing the expansion rate. In addition, the expansion rate is decreased as the foreground object is closer to the structure, so that even the foreground three-dimensional model portion is not erroneously removed. The expansion rate at this time may be linearly changed according to the distance from the foreground, or may be determined stepwise by providing one or a plurality of reference distances.

  FIG. 12A shows an integrated three-dimensional model generated based on the integrated mask, similar to FIG. 9 described above. FIG. 12B shows a three-dimensional model of the structure generated based only on the structure mask. FIG. 12C shows a three-dimensional model with only the foreground obtained by the difference between the integrated three-dimensional model in FIG. 12A and the three-dimensional model of the structure in FIG. .

  The above is the content of the three-dimensional model formation process according to the present embodiment. When generating a virtual viewpoint image of a moving image, the above-described steps are repeated for each frame to generate a three-dimensional model for each frame. However, the reception and storage of the structure mask (step 1101) and the generation of the three-dimensional model of the structure (step 1105) need only be performed immediately after the start of the flow, and the second and subsequent frames can be omitted. Furthermore, when shooting at the same shooting location with different dates and times, the reception and storage of the structure mask and the generation of the 3D model of the structure are performed for the first time and stored in the RAM or the like. You may use what you hold.

(Modification)
In the present embodiment, the three-dimensional model of the foreground is generated by subtracting the three-dimensional model of the structure from the integrated three-dimensional model of the foreground + structure. However, the present invention is not limited to this. For example, it counts which mask image is included in each voxel (or for each predetermined area) constituting the integrated 3D model of the foreground + structure, and deletes the portion whose count value is equal to or less than the threshold from the integrated 3D model. Thus, a three-dimensional model with only the foreground may be obtained. The threshold value at this time is set to an arbitrary value smaller than the total number of cameras in consideration of the installation position of each camera, the line-of-sight direction, and the like. In the case of the present embodiment in which the total number of cameras is eight and the camera is arranged as shown in FIG. 2A, only the soccer goal can be deleted by setting, for example, “2” as the threshold value.

  As described above, according to the present embodiment, it is possible to generate a highly accurate foreground-only three-dimensional model that does not include a structure even if there is a structure that hides the object that is the foreground.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

140 3D Model Generation Device 310 Data Receiving Unit 330 Mask Synthesis Unit 350 3D Model Formation Unit

Claims (22)

First acquisition means for acquiring first area information indicating areas of objects in a plurality of images obtained by shooting from a plurality of shooting directions;
Second acquisition means for acquiring second region information indicating a region of a structure that may obstruct the object during photographing from at least one of the plurality of photographing directions;
Three-dimensional shape data corresponding to the object based on both the first area information indicating the area of the object acquired by the first acquisition means and the second area information indicating the area of the structure acquired by the second acquisition means. Generating means for generating
A generation apparatus comprising: The first area information is an image indicating an area of the object,
The second area information is an image showing an area of the structure.
The generating apparatus according to claim 1, wherein: The image processing apparatus further includes a combining unit that combines an image showing the object area and an image showing the structure area,
The generating device according to claim 2, wherein the generating unit generates the three-dimensional shape data corresponding to the object based on the image combined by the combining unit.   4. The composition unit according to claim 3, wherein the composition unit generates an image showing both the object and the structure based on an image showing the object area and an image showing the structure area. Generator.   5. The generation apparatus according to claim 1, wherein the three-dimensional shape data corresponding to the object includes three-dimensional shape data corresponding to the structure. The generating means includes
Based on the second area information acquired by the second acquisition means, three-dimensional shape data corresponding to the structure is generated,
3D corresponding to the structure based on the generated 3D shape data corresponding to the structure and 3D shape data corresponding to the object including the 3D shape data corresponding to the structure The generating apparatus according to claim 5, wherein three-dimensional shape data corresponding to the object not including shape data is generated.   The generation unit generates three-dimensional shape data corresponding to the object that does not include the three-dimensional shape data corresponding to the structure, based on the three-dimensional shape data corresponding to the structure that is at least partially expanded. The generating apparatus according to claim 6.   8. The generation according to claim 7, wherein the generation unit determines a portion for expanding the three-dimensional shape data corresponding to the structure according to a region in the three-dimensional space where the structure exists. apparatus.   The generation unit determines a ratio of expanding the three-dimensional shape data corresponding to the structure according to a distance between the structure and the object in a three-dimensional space where the structure exists. Item 9. The generation device according to Item 7 or 8.   The generation device according to claim 9, wherein the generation unit increases the ratio of expanding the three-dimensional shape data corresponding to the structure as the distance between the structure and the object increases.   11. The generation according to claim 1, wherein the object is a moving body whose position can change in each image when the shooting is performed in time series from the same shooting direction. apparatus.   The generating apparatus according to claim 1, wherein the object is at least one of a person and a ball. The structure, generating device according to any one of claims 1 to 12, wherein the quiescent state is the object to continue. The generation device according to claim 1, wherein at least one of a soccer goal and a corner flag used in a soccer game is the structure. The structure, generating device according to any one of claims 1 to 14, characterized in that it is installed objects in place. 16. The generation apparatus according to claim 1, wherein at least a part of the structure is installed on a field where a person who is an object plays a game. The generation apparatus according to claim 1, wherein the structure is a designated object. A first acquisition step of acquiring first region information indicating regions of an object in a plurality of images obtained by photographing from a plurality of photographing directions;
A second acquisition step of acquiring second region information indicating a region of a structure that may block the object at the time of photographing from at least one of the plurality of photographing directions;
Three-dimensional shape data corresponding to the object based on both the first region information indicating the region of the object acquired in the first acquisition step and the second region information indicating the region of the structure acquired in the second acquisition step. A generating step for generating
A generation method characterized by comprising: The generation method according to claim 18 , wherein the three-dimensional shape data corresponding to the object includes three-dimensional shape data corresponding to the structure. The first area information is an image indicating an area of the object,
The generation method according to claim 18 or 19 , wherein the second area information is an image showing an area of the structure. And further comprising a combining step of combining the image indicating the region of the object and the image indicating the region of the structure,
The generation method according to claim 20 , wherein, in the generation step, the three-dimensional shape data corresponding to the object is generated based on the image combined in the combination step. The program for functioning a computer as a production | generation apparatus of any one of Claims 1 thru | or 17 .

Images