WO2019244944A1 - 三次元再構成方法および三次元再構成装置 - Google Patents

三次元再構成方法および三次元再構成装置 Download PDF

Info

Publication number
WO2019244944A1
WO2019244944A1 PCT/JP2019/024341 JP2019024341W WO2019244944A1 WO 2019244944 A1 WO2019244944 A1 WO 2019244944A1 JP 2019024341 W JP2019024341 W JP 2019024341W WO 2019244944 A1 WO2019244944 A1 WO 2019244944A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
dimensional model
missing
image
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/024341
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
哲史 吉川
敏康 杉尾
徹 松延
達也 小山
将貴 福田
俊介 安木
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Corp of America
Original Assignee
Panasonic Intellectual Property Corp of America
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Corp of America filed Critical Panasonic Intellectual Property Corp of America
Priority to JP2020525776A priority Critical patent/JP7285834B2/ja
Publication of WO2019244944A1 publication Critical patent/WO2019244944A1/ja
Priority to US17/072,454 priority patent/US11494975B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/001—Industrial image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/4425—Monitoring of client processing errors or hardware failure
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection

Definitions

  • the present disclosure relates to a three-dimensional reconstruction method and a three-dimensional reconstruction device.
  • Patent Literature 1 discloses a technique for distributing videos taken from a plurality of viewpoints in conjunction with movement of the viewpoint.
  • a three-dimensional model of a specific scene is generated using a group of images captured by a plurality of calibrated cameras, and a free viewpoint image (free-view image) of the scene viewed from a free viewpoint using the three-dimensional model is generated.
  • a technique for generating a viewpoint video is known.
  • the present disclosure has an object to provide a three-dimensional reconstruction method or a three-dimensional reconstruction apparatus that can notify a user that provided information has been generated from a missing three-dimensional model.
  • the three-dimensional reconstruction method is arranged at different positions from each other, a three-dimensional model using a plurality of camera images obtained from a plurality of imaging devices that image a common three-dimensional space
  • a three-dimensional reconstruction method for generating wherein the camera parameters of the plurality of imaging devices and the three-dimensional model are obtained, and based on the camera parameters and the three-dimensional model, from one imaging device to the three-dimensional model.
  • Generate a depth image indicating the distance generate a foreground image showing a region where an object is reflected in one camera image captured by the one imaging device of the plurality of camera images, By comparing the depth image and the foreground image, it is determined whether the three-dimensional model has a missing three-dimensional point. If there is missing, it outputs the missing information indicating that there is a lack of three-dimensional points on the three-dimensional model.
  • a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, and the computer program. And any combination of recording media.
  • the present disclosure can provide a three-dimensional reconstruction method or a three-dimensional reconstruction apparatus that enables a user to know that information provided is generated from a missing three-dimensional model.
  • FIG. 1 is a diagram illustrating an outline of a free viewpoint image generation system according to an embodiment.
  • FIG. 2 is a diagram illustrating a configuration of the free viewpoint image generation system according to the embodiment.
  • FIG. 3 is a block diagram of the free viewpoint image generation device according to the embodiment.
  • FIG. 4 is a block diagram of the display device according to the embodiment.
  • FIG. 5 is a diagram illustrating an operation of the free viewpoint image generation system according to the embodiment.
  • FIG. 6 is a flowchart of a three-dimensional model generation process according to the embodiment.
  • FIG. 7 is a flowchart of a process of determining missing according to the embodiment.
  • FIG. 8 is a schematic diagram for explaining a process of determining missing according to the embodiment.
  • FIG. 9 is a schematic diagram for explaining a process of filling a missing portion according to the embodiment.
  • FIG. 10 is a flowchart of a free viewpoint video generation process according to the embodiment.
  • FIG. 11 is a schematic diagram for describing a free viewpoint video generation process according to the embodiment.
  • FIG. 12 is a flowchart of a process of generating a three-dimensional model according to a modification.
  • FIG. 13 is a flowchart of a free viewpoint video generation process according to a modification.
  • the three-dimensional reconstruction method is arranged at different positions from each other, a three-dimensional model using a plurality of camera images obtained from a plurality of imaging devices that image a common three-dimensional space
  • a three-dimensional reconstruction method for generating wherein the camera parameters of the plurality of imaging devices and the three-dimensional model are obtained, and based on the camera parameters and the three-dimensional model, from one imaging device to the three-dimensional model.
  • Generate a depth image indicating the distance generate a foreground image showing a region where an object is reflected in one camera image captured by the one imaging device of the plurality of camera images, By comparing the depth image and the foreground image, it is determined whether the three-dimensional model has a missing three-dimensional point. If there is missing, it outputs the missing information indicating that there is a lack of three-dimensional points on the three-dimensional model.
  • the depth image and the foreground image are binarized, and the binarized depth image is compared with the binarized foreground image, so that the three-dimensional model has a three-dimensional structure. It may be determined whether there is a missing point.
  • the missing part of the three-dimensional model is supplemented by a three-dimensional point generated by estimating using the plurality of camera images
  • a three-dimensional model including the supplemented three-dimensional points may be output.
  • the supplemented three-dimensional point is provided with attribute information indicating that the three-dimensional point is a supplemented three-dimensional point, and the missing information is information indicated by the attribute information. There may be.
  • missing information can be easily added to the missing part of the three-dimensional model.
  • viewpoint information indicating at least one of a position and a direction of the viewpoint to the output three-dimensional model, using the three-dimensional model, the viewpoint information, the camera parameters and the plurality of camera images.
  • a composite video for displaying identification information indicating that the free viewpoint video has been generated using a three-dimensional model having a gap indicated by the gap information may be output.
  • the identification information may correspond to a missing part of the three-dimensional model on the free viewpoint video of the composite video, and may be indicated by an area displayed in a specific display mode.
  • a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, and the computer program. And any combination of recording media.
  • FIG. 1 is a diagram showing an outline of a free viewpoint image generation system.
  • a space to be photographed can be three-dimensionally reconstructed by photographing the same space from multiple viewpoints using a calibrated camera (for example, a fixed camera) (three-dimensional space reconstruction).
  • a calibrated camera for example, a fixed camera
  • three-dimensional space reconstruction By performing tracking, scene analysis, and video rendering using the three-dimensionally reconstructed data, a video viewed from an arbitrary viewpoint (free viewpoint camera) can be generated. Thereby, a next-generation wide-area monitoring system and a free viewpoint image generation system can be realized.
  • the three-dimensional model in the generation of a free viewpoint image using a three-dimensional model, the three-dimensional model generates a blind spot or a failure in generating a three-dimensional model with high accuracy.
  • the missing part is determined, and the determined missing part is specified on the free viewpoint video. As a result, it is possible to realize a system capable of accurately transmitting the missing part of the three-dimensional model to the user.
  • the user can determine that there is a blind spot area or an uncertain area where the three-dimensional model cannot be created successfully, and the user acquires erroneous information from the provided information such as a free viewpoint video. Can be reduced.
  • FIG. 2 is a diagram illustrating a configuration example of the free viewpoint image generation system 100 according to the present embodiment.
  • the free viewpoint image generation system 100 includes a plurality of cameras 101, a free viewpoint image generation device 102, and a plurality of display devices 103.
  • the plurality of cameras 101 generate a plurality of camera images (camera images) by photographing the same scene from different viewpoints. That is, the plurality of cameras 101 are arranged at positions different from each other, and image a common three-dimensional space. For example, the plurality of cameras 101 may perform synchronous shooting. Alternatively, the plurality of cameras 101 may embed time information in a video or add index information indicating a frame order to a video.
  • the plurality of cameras is an example of a plurality of imaging devices.
  • the free viewpoint image generation device 102 is a server that generates a free viewpoint image, acquires a plurality of camera images captured by a plurality of cameras 101, and uses the plurality of camera images to generate a three-dimensional model and a free viewpoint image. Generate Further, the free viewpoint image generation device 102 is connected to a plurality of display devices 103 via a network 104. The free viewpoint image generation device 102 transmits the camera video, the three-dimensional model, and the free viewpoint image to the display device 103. Specifically, the free viewpoint image generation device 102 generates a three-dimensional model and a free viewpoint image based on the time and viewpoint information transmitted from the display device 103, respectively, and generates the generated three-dimensional model and the free viewpoint image. The data is transmitted to the display device 103. At this time, the free viewpoint image generating apparatus 102 may generate a free viewpoint image using a three-dimensional model, or may generate a free viewpoint image by interpolating a video from two or more images. .
  • the display device 103 is a terminal that displays the camera image, the three-dimensional model, and the free viewpoint image transmitted from the free viewpoint image generation device 102 to the user by displaying them. Further, the display device 103 selects a time from the camera image, selects a viewpoint from the three-dimensional model, and transmits each piece of information to the free viewpoint image generation device 102.
  • the communication method is not particularly limited as long as data can be exchanged between the free viewpoint image generation device 102 and the display device 103.
  • communication between remote locations may be performed via the Internet, or communication may be performed within a LAN in a laboratory or the like.
  • the time and viewpoint for generating a free viewpoint video in the free viewpoint image generation device 102 may be determined on the free viewpoint image generation device 102 or may be determined by being specified by the display device 103.
  • FIG. 3 is a block diagram of the free viewpoint image generation device 102 according to the present embodiment.
  • the free viewpoint image generation device 102 includes a camera video reception unit 201, a three-dimensional model generation unit 202, a free viewpoint video generation unit 203, a free viewpoint video distribution unit 204, a camera video storage unit 205, and a camera parameter storage unit. 206, a three-dimensional model storage unit 207, and a free viewpoint image storage unit 208.
  • the camera video receiving unit 201 acquires a camera video from one or more cameras 101 and saves the acquired camera video in the acquired camera video storage unit 205.
  • the three-dimensional model generation unit 202 uses the camera image of the camera image storage unit 205 at the corresponding time and the camera parameters of the camera parameter storage unit 206 to perform the tertiary Generate the original model.
  • the three-dimensional model generation unit 202 stores the generated three-dimensional model in the three-dimensional model storage unit 2070.
  • the three-dimensional model generation unit 202 generates a three-dimensional model by performing three-dimensional reconstruction using an image processing technique such as SfM (Structure @ from @ Motion).
  • SfM Structure @ from @ Motion
  • the three-dimensional model generation unit 202 generates a three-dimensional model using the depth information.
  • the free viewpoint video generation unit 203 generates a free viewpoint image viewed from the viewpoint based on the viewpoint specified by the display device 103.
  • the free viewpoint video generation unit 203 stores the generated free viewpoint image in the free viewpoint image storage unit 208.
  • the free viewpoint video generation unit 203 generates a free viewpoint image using the three-dimensional model generated by the three-dimensional model generation unit 202.
  • the three-dimensional model used for generating the free viewpoint image may be a three-dimensional model stored in the three-dimensional model storage unit 207.
  • the free viewpoint video generation unit 203 generates a free viewpoint image by two-dimensionally interpolating an image between cameras as in morphing processing.
  • the free viewpoint image (free viewpoint video) may be a still image or a moving image.
  • the moving image may be a moving image showing a time series change of a specific scene viewed from a certain viewpoint, or may be a moving image in which a viewpoint position is continuously changed at a specified time. Or a combination of these.
  • the free viewpoint video distribution unit 204 receives the free viewpoint image (free viewpoint video) generated by the free viewpoint video generation unit 203 or the free viewpoint image (one or more free viewpoint videos) stored in the free viewpoint image storage unit 208. To the display device 103.
  • the camera image storage unit 205 stores the camera image captured by the camera 101. Specifically, the camera image directly obtained from the camera 101 is stored. Alternatively, the camera video storage unit 205 may store a camera video obtained indirectly via another device.
  • the camera parameter storage unit 206 stores camera parameters including the three-dimensional position and orientation information (camera pose) of the camera 101 that has captured the camera video stored in the camera video storage unit 205.
  • the three-dimensional position and orientation information is, for example, information acquired by a GPS or a gyro sensor included in the camera 101.
  • the free viewpoint image generation apparatus 102 may estimate the three-dimensional position and orientation information using an image processing technique such as SfM based on the camera video, and the camera parameter storage unit 206 Position and orientation information may be stored.
  • the three-dimensional model storage unit 207 stores the three-dimensional model generated by the three-dimensional model generation unit 202.
  • the free viewpoint image storage unit 208 stores the free viewpoint image generated by the free viewpoint video generation unit 203.
  • the camera video storage unit 205, the camera parameter storage unit 206, the three-dimensional model storage unit 207, and the free viewpoint image storage unit 208 only need to be able to store each data temporarily and for a long time
  • a storage device such as a hard disk drive (HDD) that can hold data for a long time may be used.
  • HDD hard disk drive
  • FIG. 4 is a block diagram of the display device 103 according to the present embodiment.
  • the display device 103 includes a free viewpoint video reception unit 301, a screen generation unit 302, a screen display unit 303, and a free viewpoint image storage unit 304.
  • the free viewpoint video receiving unit 301 receives the free viewpoint video generated by the free viewpoint image generation device 102, and stores the received free viewpoint video in the free viewpoint image storage unit 304.
  • the screen generation unit 302 generates a display screen for displaying the received free viewpoint image on the screen display unit 303. Further, the screen generation unit 302 may acquire operation information as a result of accepting an operation by the user, and change a free viewpoint image used to generate a display screen according to the acquired operation information.
  • the operation by the user is indicated by, for example, input to an input device such as a keyboard and a touch panel.
  • the screen display unit 303 presents the display screen generated by the screen generation unit 302 to the user by displaying the display screen.
  • the free viewpoint image storage unit 304 is a storage for storing the free viewpoint image transmitted from the free viewpoint image generation device 102.
  • the free viewpoint image storage unit 304 only needs to store temporary and long-term data, and may be a short-term storage such as a memory or a device that can store long-term data such as an HDD (hard disk drive).
  • a short-term storage such as a memory or a device that can store long-term data such as an HDD (hard disk drive).
  • HDD hard disk drive
  • FIG. 5 is a sequence diagram showing an operation of the free viewpoint image generation system 100 according to the present embodiment.
  • each of the one or more cameras 101 transmits the captured camera video to the free viewpoint image generation device 102 (S101).
  • the one or more cameras 101 are described as a camera group 101.
  • the one or more cameras 101 only need to be able to transmit a camera image to the free viewpoint image generation device 102, and control relating to image capturing such as the start of recording of the one or more cameras 101 may be performed by the free viewpoint image generation device 102. Alternatively, the control may be performed by another control device.
  • the camera video receiving unit 201 outputs the camera video obtained from the one or more cameras 101 to the three-dimensional model generating unit 202 (S102).
  • the three-dimensional model generation unit 202 acquires time information (S103).
  • the time information is, for example, information indicating a time specified by the display device 103 or another terminal.
  • the time information is information transmitted from the display device 103 or another terminal.
  • the time information may be, for example, information received from a user by an input receiving unit (not shown) of the free viewpoint image generating apparatus 102.
  • the three-dimensional model generation unit 202 generates a three-dimensional model at the corresponding time based on the time indicated by the time information (S104).
  • the three-dimensional model generation unit 202 outputs the generated three-dimensional model to the free viewpoint video generation unit 203. Details of the three-dimensional model generation processing will be described later with reference to FIG.
  • the free viewpoint video generation unit 203 acquires a camera video from the camera video reception unit 201 (S105), and acquires viewpoint information (S106).
  • the free viewpoint video generation unit 203 may acquire the camera video from the camera video storage unit 205.
  • the viewpoint information is, for example, information indicating a viewpoint specified by the display device 103 or another terminal.
  • the viewpoint information is information transmitted from the display device 103 or another terminal.
  • the viewpoint information may be, for example, information received from a user by an input receiving unit (not shown) of the free viewpoint image generation device 102.
  • the free viewpoint image generation unit 203 generates a free viewpoint image viewed from the viewpoint using the acquired three-dimensional model and camera image (S107).
  • the free viewpoint video generation unit 203 transmits the generated free viewpoint video to the display device 103. Details of the free viewpoint video generation processing will be described later with reference to FIG.
  • FIG. 6 is a flowchart of the three-dimensional model generation process (S104).
  • the three-dimensional model generation unit 202 acquires camera parameters of two or more cameras 101 with respect to camera parameters including the three-dimensional position and orientation of the camera 101, lens information, and the like (S111).
  • the three-dimensional model generation unit 202 acquires camera parameters from the camera parameter storage unit 206.
  • the three-dimensional model generation unit 202 acquires camera images of two or more cameras 101 (S112).
  • the camera images acquired at this time include the camera images of the two or more cameras 101 corresponding to the camera parameters acquired in step S111.
  • steps S111 and S112 are performed is not limited to the above, and step S112 may be performed before step S111. It suffices if a combination of camera parameters and camera images acquired from the same camera can be acquired from two or more cameras 101.
  • the three-dimensional model generation unit 202 performs three-dimensional reconstruction using the camera parameters obtained in step S111 and the camera video obtained in step S112, and generates a three-dimensional model (S113).
  • processing such as visual volume intersection, SfM (Structure from Motion) is performed.
  • the three-dimensional model generation unit 202 determines whether or not there is any missing part in the three-dimensional model generated in step S113 (S114). The details of the process of determining missing will be described later with reference to FIGS. 7 and 8.
  • the three-dimensional model generation unit 202 determines that there is a missing part in the determination of the missing part in step S114 (Yes in S115), the three-dimensional model generating unit 202 compensates for the missing part which is the missing part (S116).
  • the missing portion compensation process compensation is performed on a missing portion included in the 3D model generated in step S113 by performing an estimation process or the like using a camera image to newly generate a 3D point and supplement the 3D point. Perform processing. Thereby, the three-dimensional model generation unit 202 regenerates the three-dimensional model in which the three-dimensional points are supplemented, and outputs the regenerated three-dimensional model.
  • the three-dimensional model generation unit 202 determines that there is no loss in the determination of the loss in step S114 (No in S115), the three-dimensional model generation unit 202 outputs the three-dimensional model generated in step S113.
  • FIG. 7 is a flowchart of the missing determination process (S114).
  • FIG. 8 is a schematic diagram for explaining the determination processing of the missing part.
  • the three-dimensional model generation unit 202 acquires camera parameters used for the three-dimensional reconstruction processing in step S113 (S121).
  • the three-dimensional model generation unit 202 acquires the three-dimensional model generated in step S113 (S122). For example, as shown in FIG. 8, a three-dimensional model is generated as a result of imaging a subject 400 with a plurality of cameras 101 including one camera 101.
  • the three-dimensional model generation unit 202 generates a depth image indicating the distance from one camera 101 to the three-dimensional model based on the camera parameters and the three-dimensional model acquired in steps S121 and S122 (S123).
  • the depth image specifies the posture of one camera 101 based on the camera parameters of one camera 101 for one camera 101, and uses the specified posture and the three-dimensional model to obtain the object 400 from one camera 101.
  • a depth image including depth information indicating the distance from one camera 101 to the three-dimensional model (subject 400) when viewing the image is generated.
  • the depth image is composed of a plurality of pixels having depth information as pixel values, and the arrangement of the plurality of pixels is the same as that of a camera image obtained by one camera 101, for example.
  • the pixel value of each pixel of the depth image is a distance from one camera 101 to a point on the surface of the three-dimensional model (subject 400) specified by the corresponding pixel of the camera image obtained by one camera 101. Is shown.
  • the depth image has, for example, a larger pixel value as the distance is larger.
  • the three-dimensional model generation unit 202 acquires the camera image used for the three-dimensional reconstruction processing in step S113 (S124).
  • the three-dimensional model generation unit 202 generates a foreground image in which the region is extracted by extracting a region where the object of interest is reflected in the image from the camera image acquired in step S124.
  • the foreground image is, for example, an image in which a moving object such as a car or a truck running on a road is extracted from a camera image obtained by photographing the road.
  • the foreground image is, for example, an image obtained by capturing a background image in which an object is not reflected as a background image in advance and subtracting the background image from a camera image in which the object is reflected. That is, the foreground image is an image from which the object is extracted by deleting the background image from the camera image.
  • the foreground image may be generated by extracting an object using a technique called Semantic @ Segmentation, which gives meaning on a pixel basis.
  • the three-dimensional model generation unit 202 has a missing three-dimensional point in the three-dimensional model obtained in step S122. It is determined whether or not (S126). For example, as shown in FIG. 8, the three-dimensional model generation unit 202 binarizes the depth image and the foreground image, and compares the binarized depth image 401 with the binarized foreground image 402. Thus, it is determined whether the three-dimensional model has a missing three-dimensional point.
  • the three-dimensional model generation unit 202 sets, for example, a pixel value of a pixel having a pixel value larger than a predetermined threshold value to “1”, and sets a pixel value of a pixel having a pixel value equal to or smaller than the predetermined threshold value to “0”.
  • the depth image is binarized.
  • the three-dimensional model generation unit 202 binarizes the foreground image by, for example, setting a pixel value of a pixel having no pixel value to “1” and a pixel value of a pixel having a pixel value to “0”. I do.
  • the numerical values “1” and “0” to be set may be opposite to the above.
  • the three-dimensional model generation unit 202 obtains a subtraction result by subtracting the binarized depth image 401 from the binarized foreground image 402.
  • a missing portion image 403 having a predetermined number or more of pixels having pixel values is obtained as a subtraction result
  • the three-dimensional model generation unit 202 determines that the three-dimensional model has a missing three-dimensional point.
  • the three-dimensional model generation unit 202 determines that the three-dimensional model has no missing three-dimensional points.
  • the three-dimensional model generation unit 202 binarizes the depth image and the foreground image and compares the binarized depth image and the foreground image to determine whether the three-dimensional model has a missing three-dimensional point.
  • the three-dimensional model generation unit 202 performs grayscale or binarization on the foreground image in accordance with the range of the pixel value of the depth image, and compares the depth image with the grayscale or binarized foreground image. Thus, it may be determined whether the three-dimensional model has a missing three-dimensional point.
  • the three-dimensional model generation unit 202 determines that a missing part image having a predetermined number or more of pixels having a pixel value larger than the predetermined pixel value is obtained as a subtraction result. It is determined that there is. If the number of pixels having a pixel value larger than the predetermined pixel value is less than the predetermined number, the three-dimensional model generation unit 202 determines that the three-dimensional model has no missing three-dimensional points.
  • FIG. 9 is a schematic diagram for explaining a process for filling a missing portion.
  • the three-dimensional model generation unit 202 performs a three-dimensional reconstruction process in step S113 to obtain a camera image of the subject 400 captured by the plurality of cameras 101 and a plurality of cameras 101
  • the three-dimensional model 410 is generated by using the camera parameters of FIG.
  • the three-dimensional model generation unit 202 determines that the generated three-dimensional model 410 has the missing part 411 by the missing determination processing in step S114
  • the three-dimensional model generation unit 202 estimates the three-dimensional model 410 using the three-dimensional information of the surrounding area of the missing part 411.
  • the missing portion 411 is filled by the processing.
  • the three-dimensional model generation unit 202 When using the camera images captured by the plurality of cameras 101, the three-dimensional model generation unit 202 reduces the restriction on feature point matching (for example, NCC: Normalized ⁇ Cross ⁇ Correlation) when generating three-dimensional points, thereby reducing the three-dimensional model. A new point may be generated, and the missing portion 411 may be filled with the newly generated three-dimensional point. As a result, the three-dimensional model generation unit 202 newly generates a three-dimensional model 420 having the filling part 421 in which the missing part 411 is filled, and outputs the newly generated three-dimensional model 420.
  • the supplementing unit 421 includes a three-dimensional point generated by performing new estimation using a plurality of camera images.
  • the three-dimensional model is configured by a point cloud that is a set of a plurality of three-dimensional points, and each three-dimensional point has not only information indicating the position of the three-dimensional point, but also the color and reflectance of the three-dimensional point.
  • the three-dimensional point has attribute information such as the three-dimensional point newly added by the supplement, information indicating that the three-dimensional point is supplemented as the attribute information may be added.
  • the three-dimensional model generating unit 202 when the generated three-dimensional model has a missing three-dimensional point, the three-dimensional model generating unit 202 outputs missing information indicating that the three-dimensional model has a missing three-dimensional point.
  • the missing information may be, for example, information given to the three-dimensional model itself, or attribute information given to each of the three-dimensional points supplemented as described above.
  • the three-dimensional model generating unit 202 adds missing information indicating that there is a missing to the three-dimensional model, and outputs the three-dimensional model to which the missing information has been added. May be.
  • FIG. 10 is a flowchart of the free viewpoint video generation process (S107).
  • FIG. 11 is a schematic diagram for explaining a free viewpoint video generation process.
  • the free viewpoint video generation unit 203 acquires viewpoint information indicating at least one of the three-dimensional position of the viewpoint and the line of sight direction when generating the free viewpoint video (S131).
  • the free viewpoint video generation unit 203 acquires camera parameters of two or more cameras 101 with respect to camera parameters including the three-dimensional position and orientation of the camera 101, lens information, and the like (S132).
  • the free viewpoint video generation unit 203 acquires camera parameters from the camera parameter storage unit 206.
  • the free viewpoint video generation unit 203 acquires camera images of two or more cameras 101 (S133).
  • the camera images acquired at this time include the camera images of two or more cameras 101 corresponding to the camera parameters acquired in step S132.
  • the free viewpoint video generation unit 203 acquires the three-dimensional model generated in step S104 (S134).
  • the free viewpoint video generation unit 203 views the three-dimensional model obtained in Step S134 from the viewpoint indicated by the viewpoint information obtained in Step S131, using the camera parameters and the camera video obtained in Steps S132 and S133. A free viewpoint video at that time is generated (S135). Note that the free viewpoint video generation unit 203 projects the three-dimensional model 420 in the three-dimensional space onto the viewpoint 430 indicated by the viewpoint information, as shown in FIG. Generate 450.
  • the free viewpoint video generation unit 203 may specify the region determined to be a missing part on the free viewpoint image. For example, the free viewpoint video generation unit 203 displays a caption indicating the presence of a missing part, or displays the color of the screen frame in a color different from the free viewpoint image generated from the three-dimensional model having no missing part. By doing so, the free viewpoint video 440 indicating the presence of the missing portion may be generated. Further, the free viewpoint video generation unit 203 may generate, for example, a free viewpoint video 450 in which the location of the missing part is represented by a specific color or a specific pattern. In the free viewpoint video, an object having a missing portion may be specified.
  • the present invention is not limited to the above example and may be presented by any means as long as it is presented to the user so that the free viewpoint video is generated by the three-dimensional model having the missing part. .
  • the free viewpoint video generating unit 203 when acquiring the missing information corresponding to the three-dimensional model, the free viewpoint video generating unit 203 generates the free viewpoint video using the generated free viewpoint video and the three-dimensional model having the missing part indicated by the missing information. And a composite video for displaying the identification information indicating that the image has been output.
  • the identification information corresponds to a missing part of the three-dimensional model on the free viewpoint video and is indicated by an area displayed in a specific display mode.
  • the free viewpoint video generation unit 203 outputs the free viewpoint video 440 or the free viewpoint video 450 generated in step S135 to the display device 103 (S136).
  • the free viewpoint image generation system 100 by comparing the depth image and the foreground image, it is determined whether the 3D model has a missing 3D point, and the 3D model If the original point is missing, missing information indicating that the three-dimensional point is missing in the three-dimensional model is output. Therefore, the user can be notified that the provided information is generated from the missing three-dimensional model.
  • the free viewpoint image generation system 100 performs the missing determination process by binarizing the depth image and the foreground image, comparing the binarized depth image with the binarized foreground image, It is determined whether or not a three-dimensional point is missing in the original model. Therefore, it is possible to easily determine the lack of the three-dimensional model.
  • the free viewpoint image generation system 100 outputs the three-dimensional point generated by estimating the missing part of the three-dimensional model using a plurality of camera images when the three-dimensional model has a three-dimensional point missing. And outputs a three-dimensional model including the three-dimensional points. For this reason, information generated from the three-dimensional model in which the gap has been filled can be provided to the user.
  • the free viewpoint image generating system 100 when the missing information corresponding to the 3D model is acquired, the free viewpoint image generating system 100 generates a free viewpoint image using the generated free viewpoint image and the 3D model having the missing indicated by the missing information. And a composite video for displaying the identification information indicating that the image has been output. According to this, by outputting a synthesized video obtained by synthesizing the free viewpoint video and the identification information, the user is notified that the free viewpoint video is generated from the missing 3D model. Can be.
  • the identification information corresponds to the missing part of the three-dimensional model on the free viewpoint video of the composite video and is indicated by an area displayed in a specific display mode. For this reason, the missing part in the free viewpoint video can be presented to the user.
  • the three-dimensional model generation unit 202 determines the missing part, but the present invention is not limited to this, and the free viewpoint video generation unit 203 may perform the determination. That is, step S104a described in FIG. 12 may be performed instead of step S104 by the three-dimensional model generation unit 202. Further, step S107a described in FIG. 13 may be performed instead of step S107 by the free viewpoint video generation unit 203.
  • FIG. 12 is a flowchart of the three-dimensional model generation process (S104a) according to the modification.
  • the three-dimensional model generation process according to the modified example is different from the flowchart of FIG. 6 in that the three-dimensional model finally output by the three-dimensional model generation unit 202 does not include information on the missing part. According to this, the processing amount related to the three-dimensional model generation processing can be reduced, and the three-dimensional model can be generated at high speed.
  • FIG. 13 is a flowchart of a free viewpoint video generation process (S107a) according to the modification.
  • the free viewpoint video generation unit 203 determines whether or not there is a missing part in the three-dimensional model acquired in step S134 (S134a).
  • the missing determination process for example, the same process as in step S114 is performed. However, even when it is determined that there is a missing portion, the missing information indicating the missing portion is output to the image acquired in step S133 without performing the process of filling the missing portion as in step S116.
  • the free viewpoint video generation unit 203 views the three-dimensional model obtained in Step S134 from the viewpoint indicated by the viewpoint information obtained in Step S131, using the camera parameters and the camera video obtained in Steps S132 and S133.
  • a free viewpoint video at that time is generated (S135).
  • the free viewpoint video generation unit 203 generates a free viewpoint video by projecting, for example, a three-dimensional model 420 in a three-dimensional space to a viewpoint indicated by viewpoint information.
  • the free viewpoint video generation unit 203 specifies the region of the missing portion based on the missing information, and determines, in the free viewpoint video, a region generated using the pixel value of the region of the missing portion as a missing region.
  • the missing area may be indicated, for example, as a hatched area of the free viewpoint video 450 in FIG.
  • the free viewpoint video generation unit 203 outputs the free viewpoint video 440 or the free viewpoint video 450 generated in step S135 to the display device 103 (S136).
  • a three-dimensional reconstruction method for generating a three-dimensional model using a plurality of images obtained from a plurality of imaging devices that are arranged at different positions from each other and image a common three-dimensional space, If there is a missing three-dimensional point in the original model, missing information indicating that there is the missing may be output.
  • Each processing unit included in the free viewpoint image generation system according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include some or all of them.
  • the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor.
  • An FPGA Field Programmable Gate Array
  • a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
  • each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component.
  • Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.
  • the present disclosure may also be implemented as various methods such as a free viewpoint image generation method, a free viewpoint image generation method, or a free viewpoint image display method executed by a free viewpoint image generation system, a free viewpoint image generation device, or a display device.
  • the division of functional blocks in the block diagram is merely an example, and a plurality of functional blocks can be implemented as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be transferred to other functional blocks. You may. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or time division.
  • the present disclosure is useful as a three-dimensional reconstruction method, a three-dimensional reconstruction apparatus, or the like that can make a user know that provided information is generated from a missing three-dimensional model.
  • REFERENCE SIGNS LIST 100 free viewpoint image generation system 101 camera 102 free viewpoint image generation device 103 display device 104 network 201 camera video reception unit 202 three-dimensional model generation unit 203 free viewpoint video generation unit 204 free viewpoint video distribution unit 205 camera video storage unit 206 camera parameter Storage unit 207 3D model storage unit 208 Free viewpoint image storage unit 301 Free viewpoint video reception unit 302 Screen generation unit 303 Screen display unit 304 Free viewpoint image storage unit 400 Subject 401 Depth image 402 Foreground image 403 Missing part image 410, 420 Tertiary Original model 411 Missing part 421 Filling part 430 Perspective 440, 450 Free viewpoint video

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Radar, Positioning & Navigation (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Remote Sensing (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)
  • Image Generation (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Image Processing (AREA)
PCT/JP2019/024341 2018-06-19 2019-06-19 三次元再構成方法および三次元再構成装置 Ceased WO2019244944A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2020525776A JP7285834B2 (ja) 2018-06-19 2019-06-19 三次元再構成方法および三次元再構成装置
US17/072,454 US11494975B2 (en) 2018-06-19 2020-10-16 Method for analyzing three-dimensional model and device for analyzing three-dimensional model

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862686867P 2018-06-19 2018-06-19
US62/686,867 2018-06-19

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/072,454 Continuation US11494975B2 (en) 2018-06-19 2020-10-16 Method for analyzing three-dimensional model and device for analyzing three-dimensional model

Publications (1)

Publication Number Publication Date
WO2019244944A1 true WO2019244944A1 (ja) 2019-12-26

Family

ID=68983190

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/024341 Ceased WO2019244944A1 (ja) 2018-06-19 2019-06-19 三次元再構成方法および三次元再構成装置

Country Status (3)

Country Link
US (1) US11494975B2 (https=)
JP (1) JP7285834B2 (https=)
WO (1) WO2019244944A1 (https=)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021149192A (ja) * 2020-03-16 2021-09-27 キヤノン株式会社 画像処理装置、画像処理方法、およびプログラム
JP2023041954A (ja) * 2018-11-27 2023-03-24 キヤノン株式会社 システム及び情報処理方法
WO2023106117A1 (ja) * 2021-12-09 2023-06-15 ソニーグループ株式会社 情報処理装置、情報処理方法、及び、プログラム
WO2023135718A1 (ja) * 2022-01-14 2023-07-20 日本電信電話株式会社 ３次元モデルを作成する装置、方法及びプログラム
WO2023135717A1 (ja) * 2022-01-14 2023-07-20 日本電信電話株式会社 ３次元モデルを作成する装置、方法及びプログラム
JP2023111638A (ja) * 2022-01-31 2023-08-10 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム
JP2023553917A (ja) * 2020-12-11 2023-12-26 コーニンクレッカ フィリップス エヌ ヴェ 変動するデータを持つファイルフォーマット

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ES2881320T3 (es) * 2017-12-14 2021-11-29 Canon Kk Dispositivo de generación, procedimiento de generación y programa para modelo tridimensional
JP7608141B2 (ja) * 2020-12-14 2025-01-06 京セラ株式会社 立体表示用コントローラ

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018036898A (ja) * 2016-08-31 2018-03-08 キヤノン株式会社 画像処理装置及びその制御方法

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3681633B2 (ja) 2000-11-27 2005-08-10 日本電信電話株式会社 遠隔観戦サービスサーバ
WO2017094543A1 (ja) * 2015-12-02 2017-06-08 セイコーエプソン株式会社 情報処理装置、情報処理システム、情報処理装置の制御方法、及び、パラメーターの設定方法
US11127194B2 (en) * 2016-10-25 2021-09-21 Sony Corporation Image processing apparatus and image processing method
US10116915B2 (en) * 2017-01-17 2018-10-30 Seiko Epson Corporation Cleaning of depth data by elimination of artifacts caused by shadows and parallax
JP2019128641A (ja) * 2018-01-22 2019-08-01 キヤノン株式会社 画像処理装置、画像処理方法及びプログラム
JP7119425B2 (ja) * 2018-03-01 2022-08-17 ソニーグループ株式会社 画像処理装置、符号化装置、復号化装置、画像処理方法、プログラム、符号化方法及び復号化方法
US10789723B1 (en) * 2018-04-18 2020-09-29 Facebook, Inc. Image object extraction and in-painting hidden surfaces for modified viewpoint rendering

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018036898A (ja) * 2016-08-31 2018-03-08 キヤノン株式会社 画像処理装置及びその制御方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YUNOKI, REIJI ET AL.: "Image Segmentation in a Handheld RGB-D Movie", LECTURE PROCEEDINGS OF FIT2014 (FORUM ON INFORMATION TECHNOLOGY 2014, vol. 3, August 2014 (2014-08-01), pages 67 - 68 *

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023041954A (ja) * 2018-11-27 2023-03-24 キヤノン株式会社 システム及び情報処理方法
JP2021149192A (ja) * 2020-03-16 2021-09-27 キヤノン株式会社 画像処理装置、画像処理方法、およびプログラム
JP7506493B2 (ja) 2020-03-16 2024-06-26 キヤノン株式会社 画像処理装置、画像処理方法、およびプログラム
JP2023553917A (ja) * 2020-12-11 2023-12-26 コーニンクレッカ フィリップス エヌ ヴェ 変動するデータを持つファイルフォーマット
JP7832208B2 (ja) 2020-12-11 2026-03-17 コーニンクレッカ フィリップス エヌ ヴェ 変動するデータを持つファイルフォーマット
WO2023106117A1 (ja) * 2021-12-09 2023-06-15 ソニーグループ株式会社 情報処理装置、情報処理方法、及び、プログラム
JP7786474B2 (ja) 2022-01-14 2025-12-16 Ｎｔｔ株式会社 ３次元モデルを作成する装置、方法及びプログラム
WO2023135718A1 (ja) * 2022-01-14 2023-07-20 日本電信電話株式会社 ３次元モデルを作成する装置、方法及びプログラム
WO2023135717A1 (ja) * 2022-01-14 2023-07-20 日本電信電話株式会社 ３次元モデルを作成する装置、方法及びプログラム
JPWO2023135718A1 (https=) * 2022-01-14 2023-07-20
JP2023111638A (ja) * 2022-01-31 2023-08-10 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム
US12387420B2 (en) 2022-01-31 2025-08-12 Canon Kabushiki Kaisha Information processing apparatus, information processing method, and medium
JP7600162B2 (ja) 2022-01-31 2024-12-16 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム

Also Published As

Publication number Publication date
US20210035355A1 (en) 2021-02-04
US11494975B2 (en) 2022-11-08
JPWO2019244944A1 (ja) 2021-07-08
JP7285834B2 (ja) 2023-06-02

Similar Documents

Publication Publication Date Title
JP7285834B2 (ja) 三次元再構成方法および三次元再構成装置
US12289470B2 (en) Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
US20220116579A1 (en) Three-dimensional model distribution method and three-dimensional model distribution device
JP6669063B2 (ja) 画像処理装置および方法
US10789765B2 (en) Three-dimensional reconstruction method
US9846960B2 (en) Automated camera array calibration
EP2992508B1 (en) Diminished and mediated reality effects from reconstruction
JP7664891B2 (ja) 場面の積層深度データを生成するための方法
US20130095920A1 (en) Generating free viewpoint video using stereo imaging
JP2019075082A (ja) 深度値推定を用いた映像処理方法及び装置
JP2004235934A (ja) キャリブレーション処理装置、およびキャリブレーション処理方法、並びにコンピュータ・プログラム
CN103959770A (zh) 图像处理设备、图像处理方法和程序
GB2562037A (en) Three-dimensional scene reconstruction
CN115035235A (zh) 三维重建方法及装置
EP3757945A1 (en) Device for generating an augmented reality image
KR100560464B1 (ko) 관찰자의 시점에 적응적인 다시점 영상 디스플레이 시스템을 구성하는 방법
KR20140118083A (ko) 입체 영상 제작 시스템 및 깊이 정보 획득방법
WO2020193703A1 (en) Techniques for detection of real-time occlusion
JP2015005200A (ja) 情報処理装置、情報処理システム、情報処理方法、プログラムおよび記憶媒体
JP2019512781A (ja) 特徴追跡及びモデル登録により三次元多視点を再構成するための方法。
JP2026065296A (ja) 画像処理システムおよび画像処理方法
WO2026090498A1 (en) Methods, storage media, and systems for removing inaccuracies in monocular depth estimates
Tian et al. Upsampling range camera depth maps using high-resolution vision camera and pixel-level confidence classification
HK1182248B (en) Generating free viewpoint video using stereo imaging

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19822493

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020525776

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19822493

Country of ref document: EP

Kind code of ref document: A1