WO2019116942A1 - 3次元モデルの生成装置、生成方法、及びプログラム - Google Patents
3次元モデルの生成装置、生成方法、及びプログラム Download PDFInfo
- Publication number
- WO2019116942A1 WO2019116942A1 PCT/JP2018/044373 JP2018044373W WO2019116942A1 WO 2019116942 A1 WO2019116942 A1 WO 2019116942A1 JP 2018044373 W JP2018044373 W JP 2018044373W WO 2019116942 A1 WO2019116942 A1 WO 2019116942A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- area
- data corresponding
- dimensional shape
- shape data
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 65
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 18
- 238000003786 synthesis reaction Methods 0.000 claims abstract 3
- 238000012545 processing Methods 0.000 claims description 30
- 238000003384 imaging method Methods 0.000 claims description 25
- 238000009434 installation Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 abstract description 9
- 230000007547 defect Effects 0.000 abstract description 8
- 238000000926 separation method Methods 0.000 description 22
- 238000003860 storage Methods 0.000 description 22
- 238000010586 diagram Methods 0.000 description 20
- 238000009877 rendering Methods 0.000 description 10
- 238000006243 chemical reaction Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 230000010354 integration Effects 0.000 description 5
- 239000000284 extract Substances 0.000 description 4
- 238000007796 conventional method Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 244000025254 Cannabis sativa Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000013518 transcription Methods 0.000 description 1
- 230000035897 transcription Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/564—Depth or shape recovery from multiple images from contours
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20224—Image subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30221—Sports video; Sports image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30221—Sports video; Sports image
- G06T2207/30228—Playing field
Definitions
- the present invention relates to the generation of three-dimensional models of objects in an image.
- FIG. 1 are diagrams showing the basic principle of the visual volume intersection method. From an image obtained by photographing an object, a mask image representing a two-dimensional silhouette of the object is obtained on the imaging surface (FIG. 1A). Then, a cone that spreads in a three-dimensional space is considered so as to pass each point on the outline of the mask image from the projection center of the camera (FIG. 1 (b)).
- the three-dimensional shape (three-dimensional model) of the object is determined by finding the intersection of a plurality of viewing volumes, that is, the intersection of the viewing volumes (FIG. 1 (c)).
- shape estimation by the view volume intersection method sampling points in a space in which an object may exist are projected onto a mask image, and whether a point projected in common by a plurality of viewpoints is included in the mask image By examining, the three-dimensional shape of the object is estimated.
- the mask image needs to correctly represent the silhouette of the target object, and if the silhouette on the mask image is incorrect, the generated three-dimensional shape will also be incorrect.
- a part of a person who is a target object is interrupted by a stationary object such as a structure existing in front of the person and a part of a person's silhouette indicated by a mask image is missing, a three-dimensional image is generated There will be a defect in the model.
- the shape accuracy of the obtained three-dimensional model is degraded.
- the portion blocked by the structure is relatively small, even if it is a mask image that is partially covered by the silhouette, it is possible to obtain a three-dimensional model with high shape accuracy, so it is used as much as possible. It is desirable to do.
- the present invention has been made in view of the above problems, and an object of the present invention is to generate a three-dimensional object even if a structure or the like that obstructs a part of a target object is present in a shooting scene.
- the goal is to ensure that no defects occur in the model.
- a three-dimensional model generation apparatus includes a first mask image indicating an area of a structure which is a stationary object in each image captured at a plurality of viewpoints, and each image captured at the plurality of viewpoints Acquisition means for acquiring a second mask image indicating a foreground area that is an object of a moving object in the image, and an image photographed from the plurality of viewpoints by combining the acquired first mask image and the acquired second mask image Combining the structure and the foreground by combining means for generating a third mask image in which the area of the structure and the foreground area are integrated and the third mask image using the third mask image And generating means for generating a three-dimensional model.
- FIGS. 1 to (c) are diagrams showing the basic principle of the visual volume intersection method
- (A) is a block diagram showing the configuration of a virtual viewpoint image generation system
- (b) is a diagram showing an arrangement example of each camera constituting a camera array
- Flow chart showing the flow of three-dimensional model formation processing according to the first embodiment (A)-(h) is a figure which shows an example of the image image
- photographed with each camera (A)-(h) is a figure which shows an example of a structure mask
- A)-(h) is a figure which shows an example of a foreground mask (A)-(h) is a figure which shows an example of an integrated mask Diagram showing an example of an integrated 3D model generated based on an integrated mask
- the flowchart which shows the procedure of the processing which the 3 dimensional model formation device which relates to the execution form 3 executes A figure showing an example of a photography picture picturized with a plurality of cameras concerning Embodiment 3
- a diagram representing a three-dimensional model in the event of a dropout A figure showing an example of functional composition of a three-dimensional model generation device concerning Embodiment 4.
- the flowchart which shows the procedure of the processing which the 3 dimensional model formation device which relates to the execution form 4 executes The figure which represented the camera arrangement
- a diagram showing True / False Count according to the fourth embodiment The figure showing the functional block of the three-dimensional model generation device concerning Embodiment 5.
- FIG. 18 is a diagram showing a process flow of the three-dimensional model generation device according to the sixth embodiment.
- FIG. 18 is a diagram showing a process flow of the three-dimensional model generation device according to the sixth embodiment.
- a three-dimensional model with no or reduced defects in the foreground using a mask image that also includes a two-dimensional silhouette of a structure that blocks at least a part of the foreground in addition to the two-dimensional silhouette of the foreground in a shooting scene The aspect which produces
- a three-dimensional model including a structure that partially blocks the foreground is generated.
- “foreground” refers to imaging that can be viewed from a virtual viewpoint that moves (when its absolute position may change) when imaging is performed from the same angle in time series. It refers to a dynamic object (moving object) present in an image.
- structure refers to shooting that may block the foreground if there is no movement (the absolute position does not change, ie, it is stationary) when shooting from the same angle in chronological order
- the three-dimensional model refers to data representing a three-dimensional shape.
- the virtual viewpoint image is an image generated when the end user and / or an appointed operator freely manipulates the position and orientation of the virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image.
- the virtual viewpoint image to be generated and the multiple viewpoint images that are the origin of the virtual viewpoint image may be a moving image or a still image.
- the corner flag may be further treated as a structure, and when an indoor studio or the like is used as a shooting scene, furniture and props may be treated as a structure. That is, it may be a stationary object in which the stationary state or the near stationary state continues.
- FIG. 2A is a block diagram showing an example of the configuration of a virtual viewpoint image generation system including a three-dimensional model generation device according to the present embodiment.
- the virtual viewpoint image generation system 100 includes a camera array 110 including a plurality of cameras, a control device 120, a foreground separation device 130, a three-dimensional model generation device 140, and a rendering device 150.
- the control device 120, the foreground separation device 130, the three-dimensional model generation device 140, and the rendering device 150 generally include a CPU that performs arithmetic processing, a memory that stores results of arithmetic processing, programs, and the like. Realized by
- FIG. 2B is a view showing the arrangement of all eight cameras 211 to 218 constituting the camera array 110 in an overhead view when the field 200 is viewed from directly above.
- Each of the cameras 211 to 218 is installed to surround the field 200 at a certain height from the ground, and one of the goals is photographed from various angles to obtain multi-viewpoint image data of different viewpoints.
- a soccer court 201 is drawn (actually by a white line), and a soccer goal 202 is placed on the left side thereof.
- a cross mark 203 in front of the soccer goal 202 indicates a common line of sight direction (gaze point) of the cameras 211 to 218, and a circle 204 indicated by a broken line is an area where the cameras 211 to 218 can be photographed centering on the gaze point 203. Is shown.
- Data of multi-viewpoint images obtained by each camera of the camera array 110 is sent to the control device 120 and the foreground separation device 130.
- the cameras 211 to 218, and the control device 120 and the foreground separation device 130 are connected in a star topology, but may be in a ring or bus topology by daisy chain connection. . Also, although FIG. 2 shows an example of eight cameras, the number of cameras may be less than eight or more than eight.
- the controller 120 generates a camera parameter and a structure mask, and supplies the camera parameter and the structure mask to the three-dimensional model generator 140.
- the camera parameters include external parameters representing the position and orientation (line of sight direction) of each camera, and internal parameters representing the focal length and angle of view (shooting area) of the lens of each camera, and are obtained by calibration.
- Calibration is a process of obtaining a correspondence between a point in a three-dimensional world coordinate system acquired using a plurality of images obtained by photographing a specific pattern such as a checkerboard and a corresponding two-dimensional point.
- the structure mask is a mask image showing a two-dimensional silhouette of a structure present in each captured image acquired by each of the cameras 211 to 218.
- the mask image is a reference image for specifying where the part to be extracted in the photographed image is, and is a binary image represented by 0 and 1.
- the soccer goal 202 is treated as a structure, and a silhouette image indicating an area (two-dimensional silhouette) of the soccer goal 202 in an image captured by each camera at a predetermined angle from a predetermined position is a structure mask.
- a silhouette image indicating an area (two-dimensional silhouette) of the soccer goal 202 in an image captured by each camera at a predetermined angle from a predetermined position is a structure mask.
- what is necessary is just to use what was image
- due to the influence of sunshine fluctuations outdoors there may be cases where images taken in advance and after are inappropriate.
- a predetermined number of frames (for example, consecutive 10-second frames) of a moving image in which a player or the like appears may be used to delete the player or the like therefrom.
- a structure mask can be obtained based on an image adopting the median value of each pixel value in each frame.
- the foreground separation device 130 performs processing to discriminate a foreground area corresponding to a player or a ball on the field 200 and a background area other than that for each of the photographed images of a plurality of viewpoints to be input.
- a background image prepared in advance (which may be the same as the photographed image which is the source of the structure mask) is used. Specifically, for each captured image, the difference with the background image is obtained, and the region corresponding to the difference is specified as the foreground region. Thereby, a foreground mask indicating the foreground area for each photographed image is generated.
- a binary image in which the pixels belonging to the foreground area representing the player or the ball in the photographed image are represented by “0” and the pixels belonging to the other background areas by “1” is generated as the foreground mask. It will be done.
- the three-dimensional model generation device 140 generates a three-dimensional model of the object based on the camera parameters and the multi-viewpoint image. Details of the three-dimensional model generation device 140 will be described later.
- the data of the generated three-dimensional model is output to the rendering device 150.
- the rendering device 150 is based on the three-dimensional model received from the three-dimensional model generator 140, the camera parameters received from the control device 120, the foreground image received from the foreground separation device 130, and the background image prepared in advance. Generate Specifically, the positional relationship between the foreground image and the three-dimensional model is obtained from the camera parameters, the foreground image corresponding to the three-dimensional model is mapped, and a virtual viewpoint image is generated when the object of interest is viewed from an arbitrary angle. Be done. Thus, for example, it is possible to obtain a virtual viewpoint image of a decisive scene before the goal that the player has scored.
- the configuration of the virtual viewpoint image generation system illustrated in FIG. 2 is an example and is not limited thereto.
- one computer may have the functions of a plurality of devices (for example, the foreground separation device 130 and the three-dimensional model generation device 140).
- the module of each camera may be provided with the function of the foreground separation device 130, and the data of the photographed image and the foreground mask may be supplied from each camera.
- FIG. 3 is a functional block diagram showing an internal configuration of the three-dimensional model generation device 140 according to the present embodiment.
- the three-dimensional model generation device 140 includes a data receiving unit 310, a structure mask storage unit 320, a mask combining unit 330, a coordinate conversion unit 340, a three-dimensional model forming unit 350, and a data output unit 360. Each part will be described in detail below.
- the data reception unit 310 receives, from the control device 120, the camera parameters of the respective cameras constituting the camera array 110 and the structure mask representing the two-dimensional silhouette of the structure present in the photographed scene. Also, from the foreground separation device 130, captured images (multi-viewpoint images) obtained by each camera of the camera array 110 and foreground mask data representing a two-dimensional silhouette of the foreground present in each captured image are received.
- the structure mask is for the structure mask storage unit 320
- the foreground mask is for the mask combination unit 330
- the multi-viewpoint image is for the coordinate conversion unit 340
- the camera parameters are the coordinate conversion unit 340 and the three-dimensional model formation unit Passed to 350, respectively.
- the structure mask storage unit 320 stores and holds the structure mask in a RAM or the like, and supplies the structure mask to the mask combining unit 330 as necessary.
- the mask composition unit 330 reads out the structure mask from the structure mask storage unit 320, combines it with the foreground mask received from the data reception unit 310, and integrates both into one mask image (hereinafter referred to as “integrated mask To generate).
- integrated mask To generate The generated integrated mask is sent to the three-dimensional model formation unit 350.
- the coordinate conversion unit 340 converts the multi-viewpoint image received from the data reception unit 310 from the camera coordinate system to the world coordinate system based on the camera parameters. By this coordinate conversion, each photographed image from different viewpoints is converted into information indicating which region in the three-dimensional space is shown.
- the three-dimensional model formation unit 350 generates a three-dimensional model of an object including a structure in a photographed scene by the view volume intersection method using the multi-viewpoint image converted to the world coordinate system and the integrated mask corresponding to each camera. Do.
- the data of the generated three-dimensional model of the object is output to the rendering device 150 via the data output unit 360.
- FIG. 4 is a flowchart showing a flow of three-dimensional model formation processing according to the present embodiment. This series of processing is realized by the CPU included in the three-dimensional model generation device 140 developing a predetermined program stored in a storage medium such as a ROM or an HDD in the RAM and executing the program. Hereinafter, description will be made along the flow of FIG.
- the data receiving unit 310 displays a structure mask representing a two-dimensional silhouette of a structure (here, soccer goal 202) as viewed from each of the cameras 211 to 218, and a camera parameter of each camera. It receives from the control device 120.
- FIGS. 5A to 5H show images captured by the cameras 211 to 222 constituting the camera array 110, respectively.
- one player goal keeper
- FIGS. 5A to 5H show images captured by the cameras 211 to 222 constituting the camera array 110, respectively.
- one player (goal keeper) exists on the soccer court 201 in front of the soccer goal 202.
- image pick-up picture of Drawing 5 (a), (b), (h) since soccer goal 202 is located between a camera and a player, a part of players may be covered by soccer goal 202. There is. From each of the photographed images in FIGS.
- FIGS. 6 (a) to 6 (h) show structure masks corresponding to the photographed images of FIGS. 5 (a) to 5 (h).
- step 402 a plurality of viewpoints from which the data receiving unit 310 is the foreground mask indicating the two-dimensional silhouette of the foreground (here, the player or the ball) in the image captured by each of the cameras 211 to 218
- the image is received from the foreground separation device 130 together with the image.
- 7 (a) to 7 (h) respectively show foreground masks corresponding to the photographed images of FIGS. 5 (a) to 5 (h).
- the foreground separation device 130 extracts a region having a temporal change between images captured from the same angle as the foreground, so in each of FIGS. 7 (a), (b) and (h), the soccer goal 202 Some areas of the player hidden in the are not extracted as foreground areas.
- the received foreground mask data is sent to the mask combining unit 330.
- the mask combining unit 310 reads out the data of the structure mask from the structure mask storage unit 320, and combines the read structure mask and the foreground mask received from the data receiving unit 310. Run.
- This combination is an arithmetic processing for obtaining a logical sum (OR) for each pixel of the foreground mask and the structure mask represented by binary (black and white).
- FIGS. 8 (a) to 8 (h) respectively show the structure masks shown in FIGS. 6 (a) to (h) and the foreground masks shown in FIGS. 7 (a) to 7 (h) respectively.
- the obtained integrated mask is shown. In the finished integrated mask, no loss is seen in the silhouette of the player.
- the three-dimensional model formation unit 350 generates a three-dimensional model using the visual volume intersection method based on the integrated mask obtained at step 403.
- a model (hereinafter referred to as “integrated three-dimensional model”) representing the three-dimensional shape of the foreground and the structure present in the common imaging region between the plurality of images captured from different viewpoints is generated.
- an integrated three-dimensional model including the soccer goal 202 is generated in addition to the player and the ball.
- the generation of the integrated three-dimensional model is performed according to the following procedure. First, volume data is prepared by filling a three-dimensional space on the field 200 with a cube (voxel) having a certain size.
- the values of the voxels constituting the volume data are expressed by 0 and 1, respectively, “1” indicates a shape area, and “0” indicates a non-shape area.
- the three-dimensional coordinates of the voxel are converted from the world coordinate system to the camera coordinate system using the camera parameters (such as the installation position and the line of sight direction) of the cameras 211 to 218.
- a model is generated which represents the three-dimensional shape of the structure and the foreground by voxels.
- the three-dimensional shape may be expressed not by voxels themselves but by a set of points (point group) indicating the centers of the voxels.
- FIG. 9 shows an integrated three-dimensional model generated based on the integrated mask shown in FIG. 8, in which reference numeral 901 denotes the three-dimensional shape of the player who is the foreground, and reference numeral 902 denotes the soccer goal 202 which is a structure. It corresponds to a dimensional shape.
- reference numeral 901 denotes the three-dimensional shape of the player who is the foreground
- reference numeral 902 denotes the soccer goal 202 which is a structure. It corresponds to a dimensional shape.
- FIG. 10 shows a three-dimensional model generated using only the foreground mask according to the conventional method.
- a part of the player is not represented as a foreground area, so the part is missing in the generated three-dimensional model Resulting in.
- it is possible to avoid the occurrence of a defect in a part of the three-dimensional model of the foreground by using a mask image obtained by combining the foreground mask and the structure mask.
- the processing of each step described above is repeatedly performed in frame units to generate a three-dimensional model for each frame.
- reception and storage of the structure mask are performed only for the first time, held in the RAM, etc., and those held from the next time on are used. You may
- a three-dimensional model of a defect-free or reduced foreground is generated so as to include structures present in a photographed scene.
- an aspect of generating a three-dimensional model with no defect or reduced or reduced foreground only will be described as a second embodiment.
- the contents common to the first embodiment, such as the system configuration, will be omitted or simplified, and in the following, differences will be mainly described.
- the configuration of the three-dimensional model generation apparatus 140 of the present embodiment is also basically the same as that of the first embodiment (see FIG. 3), but differs in the following points.
- the three-dimensional model generation unit 350 First, readout of the structure mask from the structure mask storage unit 320 is performed not only by the mask combining unit 330 but also by the three-dimensional model generation unit 350.
- the dashed arrow in FIG. 3 represents this.
- the three-dimensional model generation unit 350 also generates the three-dimensional model of only the structure using the structure mask.
- the difference between the integrated three-dimensional model generated based on the integrated mask and the three-dimensional model of the structure generated based on the structure mask is calculated to obtain the three-dimensional of the foreground without defects or reduced. Extract the model
- FIG. 11 is a flowchart showing a flow of three-dimensional model formation processing according to the present embodiment. This series of processing is realized by the CPU included in the three-dimensional model generation device 140 developing a predetermined program stored in a storage medium such as a ROM or an HDD in the RAM and executing the program. Hereinafter, description will be made along the flow of FIG.
- Steps 1101 to 1104 correspond to steps 401 to 404 in the flow of FIG. 4 of the first embodiment, respectively, and the description will be omitted because there is no difference.
- the three-dimensional model formation unit 350 reads the structure mask from the structure mask storage unit 320, and generates a three-dimensional model of the structure by the visual volume intersection method.
- the three-dimensional model formation unit 350 obtains the difference between the combined three-dimensional model of the foreground + structure generated at step 1104 and the three-dimensional model of the structure generated at step 1105 Extract a 3D model.
- the three-dimensional model of the structure may be expanded by, for example, about 10% in three-dimensional space, and then the difference with the integrated three-dimensional model may be obtained. This makes it possible to reliably remove the part corresponding to the structure from the integrated three-dimensional model. At this time, only a part of the three-dimensional model of the structure may be expanded.
- the player is not allowed to inflate on the court 201 side, and is inflated only on the opposite side to the court 201.
- the portion to be inflated may be determined according to Furthermore, the expansion rate (expansion rate) may be changed depending on how far the foreground object such as the player or the ball is from the structure. For example, when the foreground object is at a position far from the structure, increasing the expansion rate ensures that the three-dimensional model of the structure is removed. Also, by decreasing the expansion rate as the foreground object is closer to the structure, the foreground three-dimensional model portion is prevented from being erroneously removed.
- the expansion rate at this time may be changed linearly according to the distance from the foreground, or may be determined stepwise by providing one or more reference distances.
- FIG. 12A shows an integrated three-dimensional model generated based on the integrated mask, as in FIG. 9 described above.
- FIG. 12 (b) shows a three-dimensional model of a structure generated based only on the structure mask.
- FIG.12 (c) has shown the three-dimensional model of only a foreground obtained by the difference of the integrated three-dimensional model of Fig.12 (a), and the three-dimensional model of the structure of FIG.12 (b). .
- the processing of each step described above is repeatedly performed in frame units to generate a three-dimensional model for each frame.
- the reception and storage of the structure mask (step 1101) and the generation of the three-dimensional model of the structure (step 1105) may be performed only immediately after the start of the flow, and may be omitted for the second and subsequent frames.
- the reception and storage of the structure mask and the three-dimensional model generation of the structure are performed only for the first time and held in the RAM etc. You may use what you hold.
- the present embodiment even if there is a structure that hides the foreground object, it is possible to generate a highly accurate three-dimensional model of the foreground only, which does not include the structure.
- the foreground-only three-dimensional model is generated by subtracting the three-dimensional model of the structure from the combined three-dimensional model of the foreground + structure. Next, it counts which mask image is included in each voxel (or each predetermined area) which composes the integrated 3D model of the foreground + structure, and deletes the part having the count value equal to or less than the threshold from the integrated 3D model
- a three-dimensional model of only the foreground is obtained is described as a third embodiment.
- the number of cameras in which the partial area is included in the foreground area indicating the area of the target object in the captured image among the plurality of cameras is first It is determined whether the condition of being less than or equal to the threshold of is satisfied.
- the first threshold an arbitrary value smaller than the number of all cameras is set in consideration of the installation position of each camera, the gaze direction, and the like. Then, a three-dimensional model of the target object including a partial region not determined to match the condition is generated.
- FIG. 13 (a) shows a single voxel of a cube.
- FIG. 13B shows a voxel set representing a target space for three-dimensional model generation.
- voxels are minute partial areas that constitute a three-dimensional space.
- FIG. 13C shows an example in which a voxel set of a three-dimensional model of a quadrangular pyramid is generated by removing voxels other than the quadrangular pyramid region from the set of FIG. 13B which is a voxel set of the target space.
- the three-dimensional space and the three-dimensional model are configured by cubic voxels will be described, but the present invention is not limited to this and may be configured by a point group or the like.
- FIG. 1 A block diagram showing a configuration example of a virtual viewpoint image generation system including a three-dimensional model generation device according to the present embodiment is the same as that shown in FIG.
- the camera array 110 is a group of imaging devices including a plurality of cameras 110a-110z, captures an object from various angles, and outputs an image to the foreground separation device 130 and the control device 120.
- the camera 110a-camera 110z, the foreground separation device 130, and the control device 120 are connected in a star topology, they may be connected in a ring topology or a bus topology by daisy chain connection.
- the camera array 110 is disposed around the stadium, for example, as shown in FIG. 14, and synchronously shoots from various angles toward a fixation point on a field common to all the cameras.
- a plurality of fixation points may be set, such as a fixation point to which half of the cameras included in the camera array 110 are directed and another fixation point to which the other half is directed.
- the foreground is a predetermined target object (subject for which a three-dimensional model is to be generated based on a photographed image) which enables viewing from an arbitrary angle in a virtual viewpoint
- a stadium Refers to the person present on the field of.
- the background is an area other than the foreground, and in the present embodiment, indicates the entire stadium (field, spectator seat, etc.).
- the foreground and background are not limited to these examples.
- the virtual viewpoint image in the present embodiment includes not only an image representing the view from the freely specifiable viewpoint, but also an entire image representing the view from the virtual viewpoint where the camera is not installed.
- the control device 120 calculates camera parameters indicating the positions and orientations of the cameras 110 a-110 z from the images captured synchronously by the camera array 110, and outputs the calculated camera parameters to the three-dimensional model generation device 140.
- the camera parameters are configured by external parameters and internal parameters.
- the external parameters are composed of a rotation matrix and a translation matrix, and indicate the position and orientation of the camera.
- the internal parameters include information such as the focal length and optical center of the camera, and indicate the angle of view of the camera, the size of the imaging sensor, and the like.
- the process of calculating camera parameters is called calibration.
- the camera parameter is, for example, a correspondence between a point in a three-dimensional world coordinate system acquired using a plurality of images obtained by photographing a specific pattern such as a checkerboard with a camera and a corresponding two-dimensional point. It can be determined by using it.
- the control device 120 calculates a structure mask image indicating a structure area that may overlap in front of the foreground in the images captured by the cameras 110a to 110z, and calculates information of the calculated structure mask image Output.
- the structure is a stationary object installed in the space to be photographed, and for example, a soccer goal is treated as a structure, and an image showing the area of the goal in the image photographed by each camera is a structure It becomes a mask image.
- the foreground separation device 130 identifies an area where a person on the field is present as a foreground and an area of other backgrounds from the images captured by a plurality of cameras input from the camera array 110, and indicates the foreground area. Output the foreground mask image.
- a method of identifying a foreground area a method of identifying an area having a difference between a background image held in advance and a photographed image as a foreground area, a method of identifying an area of a moving object as a foreground area, or the like can be used.
- the mask image is a reference image representing a specific portion to be extracted from the photographed image, and is a binary image represented by 0 and 1.
- the foreground mask image indicates an area in the photographed image where, for example, a foreground of a player or the like exists, and the image has the same resolution as that of the photographed image, with 1 representing the foreground area and 0 representing non-foreground pixels. It is.
- the format of the mask image is not limited to this, and may be information indicating the area of a specific object in the photographed image.
- the three-dimensional model generation device 140 has a function as an information processing device that generates a three-dimensional model using a plurality of captured images captured by a plurality of cameras. First, camera parameters and structure mask image information are received from the control device 120, and a foreground mask image is received from the foreground separation device 130. Then, the three-dimensional model generation device 140 integrates the structure mask image and the foreground mask image to generate an integrated mask image indicating an integrated area. Furthermore, each voxel is not included in the integrated mask image for which each voxel in the space for which a foreground three-dimensional model is to be generated is included in the integrated mask image, and each voxel is included in each of the foreground mask images.
- a three-dimensional model of the foreground is generated by, for example, the view volume intersection method, and is output to the rendering device 150.
- the rendering device 150 receives the three-dimensional model from the three-dimensional model generator 140 and receives an image representing the foreground from the foreground separation device 130. Further, the positional relationship between the image showing the foreground and the three-dimensional model is obtained from the camera parameters, and coloring is performed by pasting the foreground image corresponding to the three-dimensional model, and a virtual viewpoint image obtained by observing the three-dimensional model from an arbitrary viewpoint Generate
- the virtual viewpoint image may include an image of the background. That is, the rendering device 150 may generate a virtual viewpoint image seen from the set viewpoint by setting the background model, the foreground model, and the position of the viewpoint in the three-dimensional space. .
- the three-dimensional model generation device 140 includes a reception unit 155, a structure mask storage unit 101, a camera parameter storage unit 102, a mask integration unit 103, a coordinate conversion unit 104, a mask inside / outside determination unit 105, a threshold setting unit 106, and a foreground model generation unit 107 and an output unit 108.
- the receiving unit 155 receives, from the control device 120, a structure mask image indicating the camera parameters of each camera constituting the camera array 110 and the area of the structure.
- the receiving unit 155 also receives, from the foreground separation device 130, the images captured by the respective cameras of the camera array 110 and the foreground mask image indicating the foreground area in the image for each imaging.
- the structure mask storage unit 101 stores the structure mask image received by the reception unit 155.
- the structure mask image is a fixed image according to the position of the camera.
- the camera parameter holding unit 102 holds external parameters indicating the position and / or posture of each camera captured by the camera array 110 and internal parameters indicating the focal length and / or the image size as camera parameters.
- the mask integration unit 103 integrates the foreground mask image received from the foreground separation device 130 each time the image is taken by the camera array 110 and the structure mask image stored in the structure mask storage unit 101, and integrates the mask. Generate an image. Details of the method of integrating the foreground mask image and the structure mask image will be described later.
- the coordinate conversion unit 104 calculates the position and the angle of view of each captured image in the world coordinate system based on the camera parameters stored in the camera parameter storage unit 102, and which captured region of each captured image is in the three-dimensional space Convert to information that indicates what to indicate.
- the mask inside / outside determination unit 105 determines that the voxel is to be removed. In addition, when the number of cameras in which each voxel in the voxel space to be processed is not included in the integrated mask image is equal to or more than another threshold value, it is determined to remove the voxel.
- the threshold setting unit 106 sets each threshold for determining whether or not to remove a voxel by the mask inside / outside determining unit 105.
- the threshold may be set according to a user operation on the three-dimensional model generation device 140, or may be set automatically by the threshold setting unit 106.
- the foreground model generation unit 107 removes voxels determined to be removed by the mask inside / outside determination unit 105 among voxels in the voxel space to be processed, and a three-dimensional model is generated based on the remaining voxels.
- Generate The output unit 108 outputs the three-dimensional model generated by the foreground model generation unit 107 to the rendering device 150.
- FIG. 16 is a flowchart showing the procedure of processing performed by the three-dimensional model generation device according to this embodiment.
- step S ⁇ b> 1601 the reception unit 155 receives a structure mask image of each camera constituting the camera array 110 from the control device 120.
- FIG. 17 illustrates an example of five captured images captured by five cameras that form a part of the camera array 110.
- one person is on the field, and a goal is present as a structure on the field.
- the goal is a structure in front of the person. Because there is a part of the person is hiding.
- FIG. 18 shows a structure mask image corresponding to each captured image shown in FIG.
- the area of the goal which is a structure is shown as a binary image of 1 (white) and the area other than the structure is 0 (black).
- the reception unit 155 receives a foreground mask image indicating the foreground area from the foreground separation device 130.
- FIG. 19 shows a foreground mask image corresponding to each photographed image shown in FIG.
- the foreground separation device 130 extracts a part of the person hidden behind the goal as shown in FIGS. 19 (b), 19 (c) and 19 (d). The regions are not extracted as foreground regions. Also, in FIG. 19 (e), a part of the foot of the person who did not change with time is not extracted as the foreground region.
- the mask integration unit 103 integrates the structure mask image received in S1601 and S1602 with the foreground mask image to generate an integrated mask image.
- FIG. 20 shows an example of an integrated mask image which is the result of integrating the structure mask image shown in FIG. 18 and the foreground mask image shown in FIG.
- the integrated mask image is calculated by OR (logical sum) of a foreground mask image represented by binary values and a structure mask image.
- the mask inside / outside determination unit 105 selects one unselected voxel from the target voxel space.
- step S1605 the mask inside / outside determination unit 105 counts the number of cameras (hereinafter referred to as “false count”) in which one selected voxel is not included in the mask area of the integrated mask image of each camera.
- step S1606 the mask inside / outside determining unit 105 determines whether False Count is equal to or greater than a threshold. If False Count is greater than or equal to the threshold value, it can be determined that the selected one voxel is neither a foreground nor a structure, and the process advances to step S1607. This allows the removal of many voxels that are obviously non-foreground. On the other hand, if False Count is less than the threshold value, it can be determined that one selected voxel is a foreground or a structure, and thus the process advances to step S1608.
- step S1607 the foreground model generation unit 107 removes one selected voxel from the target voxel space.
- step S1608 the mask inside / outside determination unit 105 counts the number of cameras (hereinafter referred to as True Count) in which one selected voxel is included in the mask area of the foreground mask image of each camera.
- step S1609 the mask inside / outside determining unit 105 determines whether True Count is equal to or less than another threshold. If True Count is equal to or less than another threshold, it can be determined that one selected voxel is a structure, and thus the process advances to step S1607 to remove the selected one voxel from the object voxel space. On the other hand, if True Count exceeds the other threshold value, one selected voxel can be determined to be the foreground, so it is not removed from the object voxel space.
- the mask inside / outside determining unit 105 determines whether the processing has been completed for all voxels in the target voxel space. If the process is completed for all voxels, the process advances to step S1611. On the other hand, if the process has not been completed for all voxels, the process returns to S1604 to select the next one voxel from unselected voxels, and the same process is performed thereafter.
- step S1611 the foreground model generation unit 107 generates a three-dimensional model of the foreground using the remaining voxels after the voxel removal determination is performed on the target voxel space.
- the output unit 108 outputs the three-dimensional model of the foreground generated by the foreground model generation unit 107 to the rendering device 150.
- the above-described series of processing is performed for each frame captured by each camera.
- FIG. 21 is a view showing a voxel space of a target of three-dimensional model generation of the stadium system according to the present embodiment, and a rectangular parallelepiped area indicated by a grid represents the target voxel space.
- FIG. 22 shows the foreground when the stadium is photographed by the 16 cameras shown in FIG. 14, the foreground not detected by some cameras, the foreground hidden in a structure, the structure as a structure, and a non-foreground person,
- An example of the determination result is shown with False Count / True Count of voxels for human feet, human heads, goals, and other areas.
- one camera failed to extract the foreground of the person's foot, and three cameras hid the person's head in the goal which is a structure, and these are extracted as the foreground by the foreground separation device 130 Shall not be
- FIG. 23 is a diagram illustrating an example of a three-dimensional model generated by applying the threshold determination of False Count.
- FIG. 24 is a diagram illustrating an example of a three-dimensional model generated by applying the threshold determination of False Count and the threshold determination of TrueCount.
- FIG. 25 shows an example in which a three-dimensional model is generated by the view volume intersection method using only the foreground mask image shown in FIG. 19 (a) shows the whole person, but in the photographed images shown in FIGS. 19 (b), 19 (c) and 19 (d), part of the person's head is hidden by the goal of the structure. There is. Furthermore, in the photographed image shown in FIG. 19E, the foot of the person is not extracted as the foreground. Therefore, a part of the generated three-dimensional model is also missing.
- a camera in which a target voxel is included in a foreground mask image indicating a foreground region for each voxel in a target space to generate a three-dimensional model of a target object (foreground) It is determined whether the number of is less than or equal to a threshold (the threshold of True Count), and the voxel is removed if the number is less than or equal to the threshold.
- a threshold the threshold of True Count
- the loss of the three-dimensional model of the target object (foreground) to be generated is avoided and the quality of the three-dimensional model is improved. It can be done.
- the foreground mask image and the structure mask image are integrated to generate an integrated mask image
- the number of cameras in which the target voxel is not included in the integrated mask image is equal to or more than a threshold (a threshold of False Count).
- a threshold of False Count a threshold of False Count
- the voxels indicating the foreground are erroneously removed when outside the imaging range with many cameras.
- the number of cameras including voxels located in the area of the person located near the goal opposite to the fixation point in the imaging range is three. Yes, True Count is 3. At this time, if the threshold of True Count is 5, the voxel is removed because it is less than the threshold.
- an aspect of generating a three-dimensional model so as not to remove the foreground at a position away from the gaze point by setting the threshold based on the result of the inside / outside determination of the angle of view will be described as a fourth embodiment.
- voxels by calculating the True Count threshold value based on the number of cameras including voxels within the imaging range (within the angle of view), voxels erroneously showing the foreground even if the voxels are separated from the gaze point Avoid removing the
- the 3D model generation apparatus 140 includes a reception unit 155, a structure mask storage unit 101, a camera parameter storage unit 102, a mask integration unit 103, a coordinate conversion unit 104, a mask inside / outside determination unit 105, and a threshold setting unit 106.
- a reception unit 155 receives a signal from the base station.
- a structure mask storage unit 101 stores data from the main memory 102.
- a camera parameter storage unit 102 includes a structure mask storage unit 101, a camera parameter storage unit 102, a mask integration unit 103, a coordinate conversion unit 104, a mask inside / outside determination unit 105, and a threshold setting unit 106.
- a field angle inside / outside determination unit 109 and a threshold value calculation unit 260.
- the basic configuration of the virtual viewpoint image generation system is the same as in the first to third embodiments, and thus the description thereof is omitted.
- reception unit 155 the structure mask storage unit 101, the camera parameter storage unit 102, the mask integration unit 103, the coordinate conversion unit 104, the mask inside / outside determination unit 105, the threshold setting unit 106, which constitute the three-dimensional model generation apparatus 140.
- the foreground model generation unit 107 and the output unit 108 are also the same as in the third embodiment, so the description will be omitted.
- the angle of view inside / outside determination unit 109 determines whether each voxel in the target voxel space is within the imaging range of each camera based on the camera parameters of each camera.
- the threshold calculation unit 260 calculates a value obtained by multiplying the number of cameras determined to be within the shooting range by a predetermined ratio as a True Count threshold. For example, assuming that the number of cameras for which a certain voxel is within the imaging range is five and the predetermined ratio is 60%, the threshold value of True Count for that voxel is calculated as three.
- the threshold calculated by the threshold calculation unit 260 is output to the threshold setting unit 106, and the threshold setting unit 106 sets the threshold input from the threshold setting unit 106 as the True Count threshold.
- the threshold value may be set to a predetermined value in order to be less than.
- FIG. 27 is a flowchart showing the procedure of processing performed by the three-dimensional model generation device according to this embodiment.
- the processes of S2701 to S2704 are the same as the processes of S1601 to S1604 in the flow of FIG. 16 of the third embodiment, and thus the description thereof will be omitted.
- step S2705 the angle of view inside / outside determination unit 109 determines based on the camera parameters of each camera whether or not it is included in the angle of view of one voxel camera selected in step S2704.
- step S2706 the mask inside / outside determination unit 105 determines the number of cameras in which the selected one voxel is not included in the mask area of the integrated mask image of each camera and the selected one voxel is included in the angle of view. (Hereafter, it is called False Count).
- the threshold calculation unit 260 calculates the True Count threshold based on the number of cameras including the selected one voxel in the angle of view.
- the threshold setting unit 106 sets the True Count threshold calculated by the threshold calculation unit 260.
- FIG. 28 shows a stadium including a foreground A indicated by a black point at a close position to a gaze point indicated by a cross in the figure and a foreground B indicated by a black point at a position far from the gaze point.
- the foreground A is within the angle of view with all 16 cameras
- the foreground B is within the angle of view with only three cameras 110 k, 110 l, and 110 m.
- FIG. 29 shows an example of False Count / True Count of each of the voxel of the position of foreground A close to the fixation point and the voxel of the position of foreground B far from the fixation point in the camera arrangement shown in FIG.
- the threshold of False Count is a fixed value of 10
- the threshold of True Count is 70% of the number of cameras including voxels in the angle of view.
- the threshold value of True Count is 11.2 which is 70% of 16 units. Then, since the voxels located in the foreground A close to the fixation point are in the foreground mask image in all cameras, the True Count is 16 and the count value is equal to or greater than the threshold (11.2), so the voxels are not removed.
- Voxels at the position of foreground B far from the fixation point are out of the field angle with 13 cameras (13 cameras excluding cameras 110k, 110l and 110m), and are within the angle of view with 3 cameras (cameras 110k, 110l and 110m) Become.
- voxels are included in the integrated mask image with three cameras (cameras 110 k, 110 l, and 110 m). Therefore, the number of cameras outside the integrated mask image and within the angle of view of the voxel is zero, and False Count is zero.
- the threshold value of True Count is 2.1 which is 70% of three. Then, since the voxels located in the foreground B far from the fixation point are in the foreground mask image with three cameras, the True Count is 3 and the count value is equal to or more than the threshold (2.1), so the voxels are not removed.
- the foreground 3 which is far from the gaze point and in which the number of cameras in the angle of view is small. It is possible to generate a dimensional model. Therefore, it is possible to generate a three-dimensional model in which the drop is suppressed even in the foreground far from the fixation point.
- Embodiment 3 and Embodiment 4 described above have described the aspect of counting only cameras in which voxels are included in the foreground mask image as True Count of each voxel. However, in that case, voxels in the foreground position hidden by the structure in many cameras may be removed without the TrueCount exceeding the threshold. Therefore, an aspect of generating a three-dimensional model without omission even when the foreground is blocked by a structure with many cameras will be described as a fifth embodiment.
- the voxel may be in the foreground, so the voxel is in the structure mask image.
- the loss of the foreground is avoided by adding to the True Count a value obtained by multiplying the number of cameras determined to be included by the weight value.
- the weight value is set based on the number of cameras in which the target voxel is included in the structure mask image. Then, the value obtained by adding the number of cameras in which the target voxel is included in the foreground mask image and the value obtained by multiplying the number of cameras in which the target voxel is included in the structure mask image by the weight value is True Count If it is equal to or less than the threshold value, it is determined to remove the voxel.
- the functional configuration of the three-dimensional model generation device according to the present embodiment will be described with reference to FIG.
- the three-dimensional model generation apparatus 140 according to the present embodiment further includes a weight setting unit 300 in addition to the configuration of the three-dimensional model generation apparatus of the fourth embodiment.
- the weight setting unit 300 sets a value to be added to True Count as a weight value for one camera when it is determined that the target voxel is within the structure mask image.
- This weight value is equivalent to a value indicating the possibility of the voxel located in the foreground, and in the present embodiment, the weight value per camera is set to 0.5. Then, a value obtained by multiplying the weight value of 0.5 per camera by the number of cameras for which the target voxel is determined to be within the structure mask image is added to True Count.
- FIG. 31 is a flowchart showing the procedure of processing performed by the three-dimensional model generation apparatus according to this embodiment.
- the processes of S3101 to S3104 are the same as the processes of S2701 to S2704 in the flow of FIG. 27 of the fourth embodiment. Further, each processing of S3105 to S3108 is the same as each processing of S2706 to S2709 in the flow of FIG. 27 described above. Also, the processes of S3109 and S3110 are the same as the processes of S2705 and S2710 in the flow of FIG. 27 described above.
- step S3111 the mask inside / outside determination unit 105 counts the number of cameras in which the selected one voxel is included in the mask area of the structure mask image of each camera.
- step S 3112 the weight setting unit 300 adds a value obtained by multiplying the number of cameras included in the mask area of the structure mask image by the weight value of 0.5 per camera to the True Count calculated in step S 3108.
- the processes of S3113 to S3116 are the same as the processes of S2711 to S2714 in the flow of FIG. 27 described above. The above is the series of processes in the flow of FIG.
- FIG. 32 shows an example of True Count without weight addition and an example of True Count with weight addition according to the present embodiment in a voxel located in a certain foreground area.
- This voxel is within the angle of view for all 16 cameras, the number of cameras including the target voxel in the foreground mask image is seven, and the number of cameras including the target voxel in the structure mask image is nine It shall be a platform. In this case, there are 0 cameras (all cameras 16-7-9) whose voxels are outside the integrated mask image. Therefore, the number of cameras outside the integrated mask image and within the angle of view of the voxel is zero, and False Count is zero.
- the threshold of True Count is 70% of the number of cameras including the target voxel in the angle of view. Then, since the threshold is 11.2 (16 ⁇ 0.7), True Count (7) ⁇ threshold (11.2), and since True Count is less than or equal to the threshold, the voxel is removed. .
- weight values are set for each type of structure mask image.
- the value based on the weight value may be added to True Count.
- the electronic signboard is likely to include the foreground because it is likely to largely overlap with the foreground.
- the weight value is 0.5.
- the weight value per camera is set to 0.3. Since the electronic signboard is larger than the goal and there is no gap, it is considered that there is a high possibility of overlapping with the foreground (person), so the weight value for the electronic signboard is made larger than the weight value for the goal.
- weight values may be set according to the voxel position, the scene, the size and shape of the mask area, the area of the stadium to be photographed, and the like.
- the threshold determination is performed after the target voxel is added to the weight True Count based on the number of cameras included in the mask area of the structure mask image.
- a mode of using the number of cameras included in the structure mask image instead of the number of cameras included in the foreground mask image (True Count) used in the third embodiment will be described as a sixth embodiment.
- whether the foreground mask image is updated each time for the three-dimensional model generated based on the foreground mask image and the structure mask image, and the voxels constituting the three-dimensional model are included in the foreground mask image The processing may be complicated because of the determination. Therefore, for the three-dimensional model generated based on the foreground mask image and the structure mask image, the number of cameras included in the fixed structure mask image is counted to generate the foreground three-dimensional model not including the structure. I do.
- FIG. 33 is a diagram showing a configuration of the three-dimensional model generation device 140 in the present embodiment.
- the configuration of the three-dimensional model generation apparatus 140 in the present embodiment is substantially the same as that of the third embodiment, and description of blocks that perform the same processing will be omitted.
- the three-dimensional model generation apparatus 140 according to the present embodiment includes a mask inside / outside determination unit 3300 in place of the mask inside / outside determination unit 105.
- the mask inside / outside determination unit 3300 counts the number of cameras in which each voxel in the target voxel space is included inside and outside the integrated mask image and the structure mask image mask, and whether to remove the target voxel by the threshold determination. It is determined and output to the foreground model generation unit 107.
- the hardware configuration of the three-dimensional model generation apparatus 140 of this embodiment is the same as that of FIG.4 (b), description is abbreviate
- omitted since the hardware configuration of the three-dimensional model generation apparatus 140 of this embodiment is the same as that of FIG
- FIG. 34 is a flowchart showing the procedure of processing performed by the three-dimensional model generation device 140 in the present embodiment.
- the processes of S3401 to S3407 and S3410 to S3412 are the same as the processes of S1601 to S1607 and S1610 to S1612 described in the third embodiment with reference to FIG. To do.
- the mask inside / outside determining unit 3300 determines whether False Count is equal to or more than a threshold. If False Count is less than the threshold, it can be determined that the selected voxel is a foreground or a structure, and the process advances to step S3408.
- step S3408 the mask inside / outside determination unit 3300 counts the number of cameras (hereinafter referred to as “Structure Count”) in which pixels or areas corresponding to the selected one voxel are included in the mask area of the structure mask image of each camera. Do.
- the mask inside / outside determining unit 3300 determines whether the Structure Count is equal to or greater than a threshold. If the Structure Count is equal to or greater than the threshold, it can be determined that the selected voxel is a structure, and thus the process advances to S3407 to remove the selected voxel from the object voxel space. On the other hand, if the Structure Count is less than the threshold, the selected voxel can not be removed from the object voxel space because it can be determined to be the foreground.
- FIG. 35 shows the foreground in the virtual viewpoint image generation system shown in FIG.
- the foreground not detected by a part of cameras, the foreground hidden in a structure, the structure and a non-foreground, person, person's foot, person An example of the False Count / Structure Count of voxels and determination results for the head, goal, and other areas of
- False Count is a fixed value of 10 in the determination of S3404, False Count is 16 for voxels located in the person, feet, head, and other areas excluding the goal post of the structure, and the threshold is Because it exceeds, it is removed.
- the three-dimensional model generated by applying the threshold determination of False Count is as shown in FIG.
- the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.
- a circuit eg, an ASIC
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
Abstract
Description
図2(a)は、本実施形態に係る、3次元モデル生成装置を含む仮想視点画像生成システムの構成の一例を示すブロック図である。仮想視点画像生成システム100は、複数のカメラを含むカメラアレイ110、制御装置120、前景分離装置130、3次元モデル生成装置140、レンダリング装置150で構成される。制御装置120、前景分離装置130、3次元モデル生成装置140及びレンダリング装置150は、演算処理を行うCPU、演算処理の結果やプログラム等を記憶するメモリなどを備えた一般的なコンピュータ(情報処理装置)によって実現される。
図3は、本実施形態に係る3次元モデル生成装置140の内部構成を示す機能ブロック図である。3次元モデル生成装置140は、データ受信部310、構造物マスク保存部320、マスク合成部330、座標変換部340、3次元モデル形成部350、データ出力部360で構成される。以下、各部について詳しく説明する。
図4は、本実施形態に係る、3次元モデル形成処理の流れを示すフローチャートである。この一連の処理は、3次元モデル生成装置140が備えるCPUが、ROMやHDD等の記憶媒体にされた所定のプログラムをRAMに展開してこれを実行することで実現される。以下、図4のフローに沿って説明する。
図11は、本実施形態に係る、3次元モデル形成処理の流れを示すフローチャートである。この一連の処理は、3次元モデル生成装置140が備えるCPUが、ROMやHDD等の記憶媒体にされた所定のプログラムをRAMに展開してこれを実行することで実現される。以下、図11のフローに沿って説明する。
以上のとおり本実施形態によれば、前景となるオブジェクトを隠してしまう構造物が存在していても、構造物を含まない高精度な前景のみの3次元モデルを生成することができる。
図13(a)は立方体の単一ボクセルを示す。図13(b)は3次元モデル生成の対象空間を表したボクセル集合を示す。図13(b)に示すように、ボクセルは3次元空間を構成する微小な部分領域である。そして、図13(c)は対象空間のボクセル集合である図13(b)の集合から四角錐領域以外のボクセルを除去することで四角錐の3次元モデルのボクセル集合を生成した例を示す。なお、本実施形態では3次元空間及び3次元モデルが立方体のボクセルで構成される例を説明するが、これに限らず点群などで構成されてもよい。
本実施形態に係る3次元モデル生成装置を含む仮想視点画像生成システムの構成例を示すブロック図は、図2(a)で示すものと同じであるため、説明は省略する。
続いて、図15を参照して、本実施形態に係る3次元モデル生成装置の機能構成を説明する。3次元モデル生成装置140は、受信部155、構造物マスク保存部101、カメラパラメータ保持部102、マスク統合部103、座標変換部104、マスク内外判定部105、閾値設定部106、前景モデル生成部107及び出力部108を備えている。
本実施形態では、ボクセルを撮影範囲内(画角内)に含むカメラの台数に基づいてTrue Countの閾値を算出することにより、ボクセルが注視点から離れていたとしても、誤って前景を示すボクセルを除去してしまうことを回避する。
図26を参照して、本実施形態に係る3次元モデル生成装置の機能構成を説明する。本実施形態に係る3次元モデル生成装置140は、受信部155、構造物マスク保存部101、カメラパラメータ保持部102、マスク統合部103、座標変換部104、マスク内外判定部105、閾値設定部106、前景モデル生成部107、出力部108に加えて、画角内外判定部109及び閾値算出部260をさらに備えている。なお、仮想視点画像生成システムの基本的構成は、実施形態1~3と同様であるため、説明は省略する。また、3次元モデル生成装置140を構成する、受信部155、構造物マスク保存部101、カメラパラメータ保持部102、マスク統合部103、座標変換部104、マスク内外判定部105、閾値設定部106、前景モデル生成部107、出力部108についても、実施形態3と同じであるため説明を省略する。
図30を参照して、本実施形態に係る3次元モデル生成装置の機能構成を説明する。本実施形態に係る3次元モデル生成装置140は、実施形態4の3次元モデル生成装置の構成に加えて、重み設定部300をさらに備えている。
図33は、本実施形態における3次元モデル生成装置140の構成を示す図である。本実施形態における3次元モデル生成装置140の構成は、実施形態3とほぼ同様であり、同じ処理を行うブロックについては説明を省略する。本実施形態に係る3次元モデル生成装置140は、マスク内外判定部105に代えて、マスク内外判定部3300を備えている。マスク内外判定部3300は、対象となるボクセル空間内の各ボクセルが統合マスク画像及び構造物マスク画像のマスク内外に含まれるカメラ台数をカウントし、閾値判定により、対象となるボクセルを除去するか否かを判定し、前景モデル生成部107に出力する。また、本実施形態の3次元モデル生成装置140のハードウェア構成は、図4(b)と同様であるため、説明は省略する。
本発明は、上述の実施形態の1以上の機能を実現するプログラムを、ネットワーク又は記憶媒体を介してシステム又は装置に供給し、そのシステム又は装置のコンピュータにおける1つ以上のプロセッサーがプログラムを読出し実行する処理でも実現可能である。また、1以上の機能を実現する回路(例えば、ASIC)によっても実現可能である。
Claims (35)
- 複数の撮影方向からの撮影により得られた複数の画像内のオブジェクトの領域を示す第1領域情報を取得する第1取得手段と、
前記複数の撮影方向の少なくとも一つの撮影方向からの撮影時に前記オブジェクトを遮る可能性のある構造物の領域を示す第2領域情報を取得する第2取得手段と、
前記第1取得手段により取得したオブジェクトの領域を示す第1領域情報と前記第2取得手段により取得した構造物の領域を示す第2領域情報の両方に基づき、前記オブジェクトに対応する3次元形状データを生成する生成手段と、
を有することを特徴とする生成装置。 - 前記第1領域情報は、前記オブジェクトの領域を示す画像であり、
前記第2領域情報は、前記構造物の領域を示す画像である、
ことを特徴とする請求項1に記載の生成装置。 - 前記オブジェクトの領域を示す画像と前記構造物の領域を示す画像とを合成する合成手段をさらに有し、
前記生成手段は、前記合成手段により合成された画像に基づいて、前記オブジェクトに対応する前記3次元形状データを生成する
ことを特徴とする請求項2に記載の生成装置。 - 前記合成手段は、前記オブジェクトの領域を示す画像と前記構造物の領域を示す画像に基づき、前記オブジェクトと前記構造物の両方の領域を示す画像を生成することを特徴とする請求項3に記載の生成装置。
- 前記オブジェクトに対応する3次元形状データは、前記構造物に対応する3次元形状データを含むことを特徴とする請求項1乃至4のいずれか1項に記載の生成装置。
- 前記生成手段は、
前記第2取得手段により取得した前記第2領域情報に基づいて、前記構造物に対応する3次元形状データを生成し、
生成された前記構造物に対応する3次元形状データと、前記構造物に対応する3次元形状データを含む前記オブジェクトに対応する3次元形状データと、に基づいて、前記構造物に対応する3次元形状データを含まない前記オブジェクトに対応する3次元形状データを生成する
ことを特徴とする請求項5に記載の生成装置。 - 前記生成手段は、少なくとも一部を膨張させた前記構造物に対応する3次元形状データに基づき、前記構造物に対応する3次元形状データを含まない前記オブジェクトに対応する3次元形状データを生成することを特徴とする請求項6に記載の生成装置。
- 前記生成手段は、前記構造物が存在する3次元空間上の領域に応じて、前記構造物に対応する3次元形状データを膨張させる部分を決定することを特徴とする請求項7に記載の生成装置。
- 前記生成手段は、前記構造物が存在する3次元空間における前記構造物と前記オブジェクトの距離に応じて、前記構造物に対応する3次元形状データを膨張させる割合を決定することを特徴とする請求項7又は8に記載の生成装置。
- 前記生成手段は、前記構造物と前記オブジェクトとの距離が離れるほど、前記構造物に対応する3次元形状データを膨張させる割合を大きくすることを特徴とする請求項9に記載の生成装置。
- 前記オブジェクトは、同じ撮影方向から時系列で前記撮影を行った場合の各画像内においてその位置が変化し得る動体であることを特徴とする請求項1乃至10のいずれか1項に記載の生成装置。
- 前記オブジェクトは、人物とボールのうち少なくとも一方であることを特徴とする請求項1乃至11のいずれか1項に記載の生成装置。
- 前記構造物は、静止状態が継続する物体であることを特徴とする請求項1乃至12のいずれか1項に記載の生成装置。
- サッカーの試合に用いられるサッカーゴール及びコーナーフラッグの少なくとも一方は、前記構造物であることを特徴とする請求項1乃至12のいずれか1項に記載の生成装置。
- 前記構造物は、所定の位置に設置された物体であることを特徴とする請求項1乃至14のいずれか1項に記載の生成装置。
- 前記構造物の少なくとも一部は、オブジェクトである人物が競技を行うフィールド上に設置されていることを特徴とする請求項1乃至15のいずれか1項に記載の生成装置。
- 前記構造物は、指定された物体であることを特徴とする請求項1乃至16のいずれか1項に記載の生成装置。
- 複数の撮影方向からの撮影により得られた複数の画像内のオブジェクトの領域を示す第1領域情報を取得する第1取得工程と、
前記複数の撮影方向の少なくとも一つの撮影方向からの撮影時に前記オブジェクトを遮る可能性のある構造物の領域を示す第2領域情報を取得する第2取得工程と、
前記第1取得工程により取得したオブジェクトの領域を示す第1領域情報と前記第2取得工程により取得した構造物の領域を示す第2領域情報の両方に基づき、前記オブジェクトに対応する3次元形状データを生成する生成工程と、
を有することを特徴とする生成方法。 - 前記オブジェクトに対応する3次元形状データは、前記構造物に対応する3次元形状データを含むことを特徴とする請求項18に記載の生成方法。
- 前記第1領域情報は、前記オブジェクトの領域を示す画像であり、
前記第2領域情報は、前記構造物の領域を示す画像である
ことを特徴とする請求項18又は19に記載の生成方法。 - 前記オブジェクトの領域を示す画像と前記構造物の領域を示す画像との合成を行う合成工程をさらに有し、
前記生成工程において、前記合成工程により合成された画像に基づいて、前記オブジェクトに対応する前記3次元形状データを生成する
ことを特徴とする請求項20に記載の生成方法。 - コンピュータを、請求項1乃至17のいずれか1項に記載の生成装置として機能させるためのプログラム。
- オブジェクトに対応する3次元形状データを生成する生成装置であって、
複数の撮影方向からの撮影により得られた複数の撮影画像内の前記オブジェクトの領域、及び、前記複数の撮影方向の少なくとも一つの撮影方向からの撮影時に前記オブジェクトを遮る可能性のある構造物の領域を表す画像データを取得する第1取得手段と、
前記オブジェクトに対応する3次元形状データを生成するための3次元空間を構成する所定の要素ごとに、当該所定の要素に対応する画素又は領域を前記オブジェクトの領域に含む画像の数を取得する第2取得手段と、
前記第1取得手段により取得された画像データと、前記第2取得手段により取得された画像の数とに基づき、前記オブジェクトに対応する3次元形状データを生成する生成手段と、
を有することを特徴とする生成装置。 - 前記オブジェクトに対応する3次元形状データは、前記構造物に対応する3次元形状データを含まない前記オブジェクトに対応する3次元形状データであり、
前記生成手段は、
前記第1取得手段により取得された画像データに基づいて、前記構造物に対応する3次元形状データを含む前記オブジェクトに対応する3次元形状データを生成し、
前記構造物に対応する3次元形状データを含む前記オブジェクトに対応する3次元形状データと、前記第2取得手段により取得された画像の数とに基づいて、前記構造物に対応する3次元形状データを含まない前記オブジェクトに対応する3次元形状データを生成することを特徴とする請求項23に記載の生成装置。 - 前記構造物に対応する3次元形状データを含む前記オブジェクトに対応する3次元形状データから、前記第2取得手段により取得された画像の数が閾値以下の部分領域に対応するデータを除くことにより、前記構造物に対応する3次元形状データを含まない前記オブジェクトに対応する3次元形状データを生成することを特徴とする請求項24に記載の生成装置。
- 前記閾値は、前記複数の撮影方向に基づく値であることを特徴とする請求項25に記載の生成装置。
- 前記閾値は、前記複数の撮影方向からの撮影を行う撮影装置の設置位置に基づく値であることを特徴とする請求項25又は26に記載の生成装置。
- 前記閾値は、前記複数の撮影方向からの撮影を行う撮影装置の台数より少ない値であることを特徴とする請求項25乃至27の何れか1項に記載の生成装置。
- 前記画像データは、前記オブジェクトの領域を表す第1画像と前記構造物の領域を表す第2画像とが合成された画像データであり、
前記第2取得手段は、前記画像の数として、前記オブジェクトに対応する3次元形状データを生成するための3次元空間を構成する所定の要素ごとに、前記所定の要素に対応する画素又は領域を前記オブジェクトの領域に含む前記第1画像の数を取得することを特徴とする請求項23乃至28の何れか1項に記載の生成装置。 - 前記画像データは、前記オブジェクトの領域を表す第1画像の画像データと前記構造物の領域を表す第2画像の画像データを含み、
前記第2取得手段は、前記画像の数として、前記オブジェクトに対応する3次元形状データを生成するための3次元空間を構成する所定の要素ごとに、前記所定の要素に対応する画素又は領域を前記オブジェクトの領域に含む前記第1画像の数を取得することを特徴とする請求項23乃至28の何れか1項に記載の生成装置。 - オブジェクトに対応する3次元形状データを生成する生成装置であって、
複数の撮影方向からの撮影により得られた複数の撮影画像内の前記オブジェクトの領域、及び、前記複数の撮影方向の少なくとも一つの撮影方向からの撮影時に前記オブジェクトを遮る可能性のある構造物の領域を表す画像データを取得する第1取得手段と、
前記オブジェクトに対応する3次元形状データを生成するための3次元空間を構成する所定の要素ごとに、当該所定の要素に対応する画素又は領域を前記構造物の領域に含む画像の数を取得する第2取得手段と、
前記第1取得手段により取得された画像データと、前記第2取得手段により取得された画像の数とに基づき、前記オブジェクトに対応する3次元形状データを生成する生成手段と、
を有することを特徴とする生成装置。 - 前記画像データは、前記オブジェクトの領域を表す第1画像と前記構造物の領域を表す第2画像とが合成された画像データであり、
前記第2取得手段は、前記画像の数として、前記オブジェクトに対応する3次元形状データを生成するための3次元空間を構成する所定の要素ごとに、当該所定の要素に対応する画素又は領域を前記構造物の領域に含む前記第2画像の数を取得することを特徴とする請求項31に記載の生成装置。 - 前記画像データは、前記オブジェクトの領域を表す第1画像の画像データと前記構造物の領域を表す第2画像の画像データを含み、
前記第2取得手段は、前記画像の数として、前記オブジェクトに対応する3次元形状データを生成するための3次元空間を構成する所定の要素ごとに、当該所定の要素に対応する画素又は領域を前記構造物の領域に含む前記第2画像の数を取得することを特徴とする請求項31に記載の生成装置。 - 前記第1画像と前記第2画像は、前記生成装置が有する受信手段を介して取得されることを特徴とする請求項29、30、32及び33の何れか1項に記載の生成装置。
- 前記要素は、前記3次元空間を構成する点又はボクセルであることを特徴とする請求項23乃至34の何れか1項に記載の生成装置。
Priority Applications (12)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201880028645.2A CN110622215B (zh) | 2017-12-14 | 2018-12-03 | 三维模型的生成装置、生成方法和程序 |
ES18887303T ES2881320T3 (es) | 2017-12-14 | 2018-12-03 | Dispositivo de generación, procedimiento de generación y programa para modelo tridimensional |
AU2018384159A AU2018384159B2 (en) | 2017-12-14 | 2018-12-03 | Generation device, generation method and program for three-dimensional model |
EP21173382.9A EP3901910A1 (en) | 2017-12-14 | 2018-12-03 | Generation device, generation method and program for three-dimensional model |
KR1020197032124A KR102178239B1 (ko) | 2017-12-14 | 2018-12-03 | 3차원 모델의 생성 장치, 생성 방법, 및 프로그램 |
RU2019134216A RU2744699C1 (ru) | 2017-12-14 | 2018-12-03 | Устройство формирования, способ формирования и программа для трехмерной модели |
BR112019022798-9A BR112019022798A2 (pt) | 2017-12-14 | 2018-12-03 | Dispositivo de geração, método de geração e programa para modelo tridimensional |
CA3061699A CA3061699C (en) | 2017-12-14 | 2018-12-03 | Generation device, generation method and non-transitory computer-readable meadium storing a program for generating a three-dimensional model from an image |
EP18887303.8A EP3608873B1 (en) | 2017-12-14 | 2018-12-03 | Generation device, generation method and program for three-dimensional model |
KR1020207032192A KR102254407B1 (ko) | 2017-12-14 | 2018-12-03 | 3차원 모델의 생성 장치, 생성 방법, 및 프로그램 |
US16/663,019 US11151787B2 (en) | 2017-12-14 | 2019-10-24 | Generation device, generation method and storage medium for three-dimensional model from object images and structure images |
US17/477,374 US11798233B2 (en) | 2017-12-14 | 2021-09-16 | Generation device, generation method and storage medium for three-dimensional model that remove a portion of the three-dimensional model |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2017239891A JP6403862B1 (ja) | 2017-12-14 | 2017-12-14 | 3次元モデルの生成装置、生成方法及びプログラム |
JP2017-239891 | 2017-12-14 | ||
JP2018-089467 | 2018-05-07 | ||
JP2018089467 | 2018-05-07 | ||
JP2018-209196 | 2018-11-06 | ||
JP2018209196A JP7265825B2 (ja) | 2018-05-07 | 2018-11-06 | 生成装置、生成方法およびプログラム |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/663,019 Continuation US11151787B2 (en) | 2017-12-14 | 2019-10-24 | Generation device, generation method and storage medium for three-dimensional model from object images and structure images |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019116942A1 true WO2019116942A1 (ja) | 2019-06-20 |
Family
ID=66820283
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2018/044373 WO2019116942A1 (ja) | 2017-12-14 | 2018-12-03 | 3次元モデルの生成装置、生成方法、及びプログラム |
Country Status (3)
Country | Link |
---|---|
US (1) | US11798233B2 (ja) |
AU (1) | AU2018384159B2 (ja) |
WO (1) | WO2019116942A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2021081603A1 (en) * | 2019-11-01 | 2021-05-06 | Visionary Machines Pty Ltd | Systems and methods for generating and/or using 3-dimensional information with camera arrays |
WO2022019149A1 (ja) * | 2020-07-21 | 2022-01-27 | ソニーグループ株式会社 | 情報処理装置、3dモデル生成方法、情報処理方法およびプログラム |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020242047A1 (en) * | 2019-05-30 | 2020-12-03 | Samsung Electronics Co., Ltd. | Method and apparatus for acquiring virtual object data in augmented reality |
JP2021060252A (ja) * | 2019-10-04 | 2021-04-15 | 三菱重工業株式会社 | 画像測定方法及び画像測定システム |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH11149576A (ja) * | 1997-09-09 | 1999-06-02 | Sanyo Electric Co Ltd | 3次元モデリング装置、3次元モデリング方法、3次元モデリングプログラムを記録した媒体、および3次元形状推定プログラムを記録した媒体 |
JP2008191072A (ja) * | 2007-02-07 | 2008-08-21 | Nippon Telegr & Teleph Corp <Ntt> | 3次元形状復元方法,3次元形状復元装置,その方法を実装した3次元形状復元プログラム及びそのプログラムを記録した記録媒体 |
JP2012088114A (ja) * | 2010-10-18 | 2012-05-10 | Topcon Corp | 光学情報処理装置、光学情報処理方法、光学情報処理システム、光学情報処理プログラム |
JP2014010805A (ja) | 2012-07-03 | 2014-01-20 | Nippon Telegr & Teleph Corp <Ntt> | 画像処理装置、画像処理方法及び画像処理プログラム |
JP2018089467A (ja) | 2013-12-09 | 2018-06-14 | ビエン−エアー ホールディング エスアーBien−Air Holding SA | 歯科用又は外科用圧縮空気ハンドピース及び該ハンドピースのためのタービン |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040104935A1 (en) * | 2001-01-26 | 2004-06-03 | Todd Williamson | Virtual reality immersion system |
EP1862969A1 (en) | 2006-06-02 | 2007-12-05 | Eidgenössische Technische Hochschule Zürich | Method and system for generating a representation of a dynamically changing 3D scene |
GB2455313B (en) | 2007-12-04 | 2012-06-13 | Sony Corp | Apparatus and method for estimating orientation |
US9171403B2 (en) * | 2014-02-13 | 2015-10-27 | Microsoft Technology Licensing, Llc | Contour completion for augmenting surface reconstructions |
US9852238B2 (en) * | 2014-04-24 | 2017-12-26 | The Board Of Trustees Of The University Of Illinois | 4D vizualization of building design and construction modeling with photographs |
KR20180092495A (ko) * | 2017-02-09 | 2018-08-20 | 한국전자통신연구원 | 다중 카메라 영상 내 관심 객체 중심의 베스트 뷰 영상 생성 장치 및 방법 |
-
2018
- 2018-12-03 WO PCT/JP2018/044373 patent/WO2019116942A1/ja unknown
- 2018-12-03 AU AU2018384159A patent/AU2018384159B2/en active Active
-
2021
- 2021-09-16 US US17/477,374 patent/US11798233B2/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH11149576A (ja) * | 1997-09-09 | 1999-06-02 | Sanyo Electric Co Ltd | 3次元モデリング装置、3次元モデリング方法、3次元モデリングプログラムを記録した媒体、および3次元形状推定プログラムを記録した媒体 |
JP2008191072A (ja) * | 2007-02-07 | 2008-08-21 | Nippon Telegr & Teleph Corp <Ntt> | 3次元形状復元方法,3次元形状復元装置,その方法を実装した3次元形状復元プログラム及びそのプログラムを記録した記録媒体 |
JP2012088114A (ja) * | 2010-10-18 | 2012-05-10 | Topcon Corp | 光学情報処理装置、光学情報処理方法、光学情報処理システム、光学情報処理プログラム |
JP2014010805A (ja) | 2012-07-03 | 2014-01-20 | Nippon Telegr & Teleph Corp <Ntt> | 画像処理装置、画像処理方法及び画像処理プログラム |
JP2018089467A (ja) | 2013-12-09 | 2018-06-14 | ビエン−エアー ホールディング エスアーBien−Air Holding SA | 歯科用又は外科用圧縮空気ハンドピース及び該ハンドピースのためのタービン |
Non-Patent Citations (2)
Title |
---|
IWASHITA, YUMI ET AL.: "3D Tracking of Multiple moving objects using fast level set method", JOURNAL OF THE ROBOTICS SOCIETY OF JAPAN, vol. 23, no. 7, 1 October 2005 (2005-10-01), pages 47 - 54, XP055640845, DOI: 10.7210/jrsj.23.813 * |
LAURENTINI A: "The Visual Hull Concept for Silhouette-Based Image Understanding", IEEE TRANSCRIPTIONS PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 16, no. 2, February 1994 (1994-02-01), pages 150 - 162, XP008103532, DOI: 10.1109/34.273735 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2021081603A1 (en) * | 2019-11-01 | 2021-05-06 | Visionary Machines Pty Ltd | Systems and methods for generating and/or using 3-dimensional information with camera arrays |
WO2022019149A1 (ja) * | 2020-07-21 | 2022-01-27 | ソニーグループ株式会社 | 情報処理装置、3dモデル生成方法、情報処理方法およびプログラム |
Also Published As
Publication number | Publication date |
---|---|
US11798233B2 (en) | 2023-10-24 |
US20220005276A1 (en) | 2022-01-06 |
AU2018384159B2 (en) | 2021-02-11 |
AU2018384159A1 (en) | 2019-11-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
KR102178239B1 (ko) | 3차원 모델의 생성 장치, 생성 방법, 및 프로그램 | |
WO2019116942A1 (ja) | 3次元モデルの生成装置、生成方法、及びプログラム | |
US11354840B2 (en) | Three dimensional acquisition and rendering | |
US10789765B2 (en) | Three-dimensional reconstruction method | |
JP6513169B1 (ja) | 仮想視点画像を生成するシステム、方法及びプログラム | |
KR20190034092A (ko) | 화상처리 장치, 화상처리 방법, 화상처리 시스템 및 기억매체 | |
JP5725953B2 (ja) | 撮像装置及びその制御方法、並びに情報処理装置 | |
WO2018135510A1 (ja) | 三次元再構成方法及び三次元再構成装置 | |
JP7170224B2 (ja) | 三次元生成方法および三次元生成装置 | |
JP6403862B1 (ja) | 3次元モデルの生成装置、生成方法及びプログラム | |
JP2019045991A (ja) | 生成装置、生成方法、及びプログラム | |
WO2018052100A1 (ja) | 画像処理装置、画像処理方法、画像処理プログラム | |
KR20220121533A (ko) | 어레이 카메라를 통해 획득된 영상을 복원하는 영상 복원 방법 및 영상 복원 장치 | |
CN113647093A (zh) | 图像处理装置、3d模型生成方法和程序 | |
WO2019211970A1 (ja) | 三次元再構成方法及び三次元再構成装置 | |
JP2019106170A (ja) | 3次元モデルの生成装置、生成方法及びプログラム | |
JP7265825B2 (ja) | 生成装置、生成方法およびプログラム | |
JP2019114269A (ja) | 仮想視点画像を生成するシステム、方法及びプログラム | |
JP7353527B2 (ja) | 情報処理装置、情報処理装置の制御方法およびプログラム | |
JP2023026244A (ja) | 画像生成装置および画像生成方法、プログラム | |
JP2022094788A (ja) | 生成装置、生成方法、及び、プログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18887303 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20197032124 Country of ref document: KR Kind code of ref document: A |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112019022798 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 2018384159 Country of ref document: AU Date of ref document: 20181203 Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2018887303 Country of ref document: EP Effective date: 20191106 |
|
ENP | Entry into the national phase |
Ref document number: 112019022798 Country of ref document: BR Kind code of ref document: A2 Effective date: 20191030 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |