WO2020024684A1 - 三维场景建模方法及装置、电子装置、可读存储介质及计算机设备 - Google Patents

三维场景建模方法及装置、电子装置、可读存储介质及计算机设备 Download PDF

Info

Publication number
WO2020024684A1
WO2020024684A1 PCT/CN2019/088550 CN2019088550W WO2020024684A1 WO 2020024684 A1 WO2020024684 A1 WO 2020024684A1 CN 2019088550 W CN2019088550 W CN 2019088550W WO 2020024684 A1 WO2020024684 A1 WO 2020024684A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
scene
dimensional
information
visible light
Prior art date
Application number
PCT/CN2019/088550
Other languages
English (en)
French (fr)
Inventor
程杰
陈岩
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2020024684A1 publication Critical patent/WO2020024684A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/257—Colour aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals

Definitions

  • the present application relates to the field of three-dimensional modeling technology, and in particular, to a three-dimensional scene modeling method, a three-dimensional scene modeling device, an electronic device, a non-volatile computer-readable storage medium, and a computer device.
  • the existing three-dimensional scene modeling usually uses a depth camera to capture a depth image of the scene and a visible light camera to capture a two-dimensional visible light image.
  • the depth information of the depth image and the two-dimensional visible light image color information are used to three-dimensionally model the scene.
  • the three-dimensional scene modeling device includes a first acquisition module, a second acquisition module, a processing module, a calculation module, and a construction module.
  • the first acquisition module is configured to acquire a depth image of the scene.
  • the second acquisition module is configured to acquire a visible light image of the scene.
  • the processing module is configured to process the depth image and the visible light image to identify an occluded object in the scene and a category of the occluded object.
  • the calculation module is configured to calculate the estimated depth information and the estimated color information of the occluded object according to the measured depth information indicated by the depth image, the measured color information indicated by the visible light image, and the category.
  • the construction module is configured to construct a three-dimensional color model of the scene according to the measured depth information, the measured color information, the estimated depth information, and the estimated color information.
  • the electronic device includes a depth camera, a visible light camera, and a processor.
  • the depth camera is used to collect a depth image of the scene.
  • the visible light camera is configured to collect a visible light image of the scene.
  • the processor is configured to process the depth image and the visible light image to identify an occluded object and a category of the occluded object in the scene, and according to the measurement depth information indicated by the depth image and the measurement indicated by the visible light image Color information and the category to calculate estimated depth information and estimated color information of the occluded object, and construct the scene based on the measured depth information, the measured color information, the estimated depth information, and the estimated color information Three-dimensional color model.
  • the computer device includes a memory and a processor.
  • the memory stores computer-readable instructions.
  • the processor causes the processor to execute the foregoing three-dimensional scene modeling method.
  • FIG. 2 is a schematic block diagram of a three-dimensional scene modeling apparatus according to some embodiments of the present application.
  • FIG. 4 is a schematic flowchart of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 5 is a schematic flowchart of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 6 is a schematic block diagram of a three-dimensional scene modeling apparatus according to some embodiments of the present application.
  • FIG. 8 is a schematic diagram of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 9 is a schematic flowchart of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 10 is a schematic flowchart of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 11 is a schematic block diagram of a three-dimensional scene modeling apparatus according to some embodiments of the present application.
  • FIG. 12 is a schematic block diagram of a processing unit of a three-dimensional scene modeling apparatus according to some embodiments of the present application.
  • FIG. 13 is a schematic flowchart of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 14 is a schematic block diagram of a calculation module of a three-dimensional scene modeling device according to some embodiments of the present application.
  • FIG. 15 is a schematic flowchart of a three-dimensional scene modeling method according to some embodiments of the present application.
  • FIG. 17 is a schematic block diagram of a computer device according to some embodiments of the present application.
  • the three-dimensional scene modeling method includes: collecting a depth image of the scene; collecting a visible light image of the scene; processing the depth image and the visible light image to identify occluded objects and the category of the occluded objects in the scene; measured depth information indicated by the depth image and visible light image indication Calculate the estimated depth information and estimated color information of the occluded object based on the measured color information and category; build a three-dimensional color model of the scene based on the measured depth information, measured color information, estimated depth information, and estimated color information.
  • the depth image includes multiple images, and the multiple depth images have different shooting angles.
  • the step of processing the depth image and the visible light image to identify occluded objects and types of occluded objects in the scene It also includes: stitching a plurality of the depth images to obtain a wide-angle depth image of the scene.
  • stitching multiple depth images to obtain a wide-angle depth image of the scene includes: determining a reference coordinate system; converting measured depth information into unified depth information in the reference coordinate system; and making a depth image according to the unified depth information. To obtain a wide-angle depth image.
  • the step of processing a wide-angle depth image and a wide-angle visible light image to identify occluded objects and categories of occluded objects in the scene includes: processing the wide-angle depth image and wide-angle visible light image to extract occluded objects;
  • the two-dimensional object model library corresponding to the occluded object is found in the two-dimensional object model library of the two-dimensional object model.
  • the category of the two-dimensional object model is the category of the occluded object.
  • the step of calculating the estimated depth information and estimated color information of the occluded object according to the measured depth information indicated by the depth image and the measured color information and category indicated by the visible light image includes: obtaining the occluded object according to the unified depth information and category Size information; find a 3D object modeling method corresponding to the category of the occluded object in a 3D object modeling method library including multiple 3D object modeling methods, a plurality of 3D object modeling methods and a plurality of 2D object models One correspondence; calculate the estimated depth information of the occluded object based on the size information, coordinate information corresponding to the unified depth information and the three-dimensional object modeling method; calculate the occluded object based on the measured color information of the occluded object and the two-dimensional object model corresponding to the occluded object Estimate color information.
  • the step of constructing a three-dimensional color model of the scene according to the measured depth information, the measured color information, the estimated depth information, and the estimated color information includes: constructing a three-dimensional model of the scene according to the unified depth information and the estimated depth information; Color information and estimated color information are used to map a three-dimensional model to obtain a three-dimensional color model.
  • the stitching module includes a determining unit 141, a converting unit 142, and a stitching unit 143.
  • the determining unit 141 is configured to determine a reference coordinate system.
  • the conversion unit 142 is configured to convert the measured depth information into unified depth information in a reference coordinate system.
  • the stitching unit 143 is configured to stitch the depth images according to the unified depth information to obtain a wide-angle depth image.
  • the visible light image includes multiple images, the multiple visible light images have different shooting angles, and the multiple visible light images correspond to the multiple depth images one by one.
  • the stitching module 14 is also used to stitch multiple visible light images to obtain a wide-angle visible light image of the scene.
  • the processing unit 151 includes a processing sub-unit 1511 and a searching sub-unit 1512.
  • the processing subunit 1511 is configured to process a wide-angle depth image and a wide-angle visible light image to extract occluded objects.
  • the finding subunit 1512 is configured to find a two-dimensional object model corresponding to the occluded object from a two-dimensional object model library including a plurality of categories of two-dimensional object models.
  • the category of the two-dimensional object model is the category of the occluded object.
  • the calculation module includes an obtaining unit 171, a searching unit 172, a first calculation unit 173, and a second calculation unit 174.
  • the obtaining unit 171 is configured to obtain the size information of the occluded object according to the unified depth information and the category.
  • the finding unit 172 is used to find a three-dimensional object modeling method corresponding to the category of the occluded object in a three-dimensional object modeling method library including a plurality of three-dimensional object modeling methods, a plurality of three-dimensional object modeling methods and a plurality of two-dimensional object models. One-to-one correspondence.
  • the first calculation unit 173 is configured to calculate the estimated depth information of the occluded object according to the size information, coordinate information corresponding to the unified depth information, and a three-dimensional object modeling method.
  • the second calculation unit 174 is configured to calculate the estimated color information of the occluded object according to the measured color information of the occluded object and the two-dimensional object model corresponding to the occluded object.
  • the building module includes a building unit 191 and a mapping unit 192.
  • the constructing unit 191 is configured to construct a three-dimensional model of the scene according to the unified depth information and the estimated depth information.
  • the mapping unit 192 is configured to map the three-dimensional model according to the measured color information and the estimated color information to obtain a three-dimensional color model.
  • the present application further provides an electronic device 100.
  • the electronic device 100 includes a depth camera 20, a visible light camera 30, and a processor 40.
  • the depth camera 20 is configured to collect a depth image of the scene.
  • the visible light camera 30 is used to collect a visible light image of a scene.
  • the processor 40 is configured to process the depth image and the visible light image to identify the occluded object and the category of the occluded object in the scene, and calculate the estimated depth information of the occluded object according to the measured depth information indicated by the depth image, the measured color information and the category indicated by the visible light image. And estimated color information, and build a three-dimensional color model of the scene based on the measured depth information, the measured color information, the estimated depth information, and the estimated color information.
  • the visible light image includes multiple images, the multiple visible light images have different shooting angles, and the multiple visible light images correspond to the multiple depth images one by one.
  • the processor 40 is further configured to stitch multiple visible light images to obtain a wide-angle visible light image of the scene.
  • the processor 40 is further configured to process a wide-angle depth image and a wide-angle visible light image to identify occluded objects and types of occluded objects in the scene.
  • the processor 40 is further configured to: obtain the size information of the occluded object according to the unified depth information and the category; and a 3D object modeling method library including a plurality of 3D object modeling methods.
  • a 3D object modeling method library including a plurality of 3D object modeling methods.
  • To find three-dimensional object modeling methods corresponding to the types of occluded objects multiple three-dimensional object modeling methods correspond one-to-one to multiple two-dimensional object models; coordinate information corresponding to size information, unified depth information, and three-dimensional object modeling The method calculates the estimated depth information of the occluded object; calculates the estimated color information of the occluded object according to the measured color information of the occluded object and the two-dimensional object model corresponding to the occluded object.
  • the processor 40 is further configured to: construct a three-dimensional model of the scene according to the unified depth information and the estimated depth information; map the three-dimensional model according to the measured color information and the estimated color information to obtain Three-dimensional color model.
  • the application also provides a non-volatile computer-readable storage medium containing computationally executable instructions.
  • the processors are caused to execute the three-dimensional scene modeling method according to any one of the foregoing embodiments.
  • the present application further provides a computer device 200.
  • the computer device 200 includes a memory 220 and a processor 210.
  • Computer-readable instructions 230 are stored in the memory 220.
  • the processor 210 is caused to execute the three-dimensional scene modeling method according to any one of the foregoing embodiments.
  • S5 Process the depth image and visible light image to identify occluded objects and the types of occluded objects in the scene;
  • S9 Construct a three-dimensional color model of the scene according to the measured depth information, the measured color information, the estimated depth information, and the estimated color information.
  • the present application further provides a three-dimensional scene modeling device 10.
  • the three-dimensional scene modeling method according to the embodiment of the present application may be implemented by the three-dimensional scene modeling apparatus 10 according to the embodiment of the present application.
  • the three-dimensional scene modeling device 10 includes a first acquisition module 11, a second acquisition module 13, a processing module 15, a calculation module 17, and a construction module 19.
  • Step S1 can be implemented by the first acquisition module 11
  • step S3 can be implemented by the second acquisition module 13
  • step S5 can be implemented by the processing module
  • step S7 can be implemented by the calculation module 17,
  • step S9 can be implemented by the construction module 19.
  • the first acquisition module 11 can be used to acquire a depth image of a scene.
  • the second acquisition module 13 may be configured to acquire a visible light image of the scene.
  • the processing module 15 may be configured to process the depth image and the visible light image to identify occluded objects and types of occluded objects in the scene.
  • the calculation module 17 may be configured to calculate the estimated depth information and the estimated color information of the occluded object according to the measured depth information indicated by the depth image, the measured color information indicated by the visible light image, and the category.
  • the construction module 19 may be configured to construct a three-dimensional color model of the scene according to the measured depth information, the measured color information, the estimated depth information, and the estimated color information.
  • the present application further provides an electronic device 100.
  • the three-dimensional scene modeling method according to the embodiment of the present application may also be implemented by the electronic device 100 according to the embodiment of the present application.
  • the electronic device 100 includes a depth camera 20, a visible light camera 30, and a processor 40.
  • Step S1 may be implemented by the depth camera 20.
  • Step S3 may be implemented by the visible light camera 30, and steps S5, S7, and S9 may all be implemented by the processor 40.
  • the depth camera 20 can be used to acquire a depth image of a scene.
  • the visible light camera 30 may be used to collect a visible light image of a scene.
  • the processor 40 may be configured to process the depth image and the visible light image to identify the occluded object and the category of the occluded object in the scene, and calculate the estimated depth information of the occluded object based on the measured depth information indicated by the depth image, the measured color information and the category indicated by the visible light image, and Estimate color information, and build a three-dimensional color model of the scene based on measured depth information, measured color information, estimated depth information, and estimated color information.
  • the electronic device 100 may be a smart phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart helmet, smart glasses, etc.).
  • a smart wearable device such as a smart watch, a smart bracelet, a smart helmet, smart glasses, etc.
  • the embodiment of the present application uses the electronic device 100 as a mobile phone as an example for description. It can be understood that the specific form of the electronic device 100 may be other, and is not limited herein.
  • the depth camera 20 may be a structured light depth camera.
  • the depth camera 20 includes a structured light projector and an infrared camera.
  • the structured light projector is used to project laser speckles into the scene, and the infrared camera collects the laser pattern modulated by the objects in the scene.
  • the processor 40 can calculate the depth information of the scene based on the offset between the laser pattern and the pre-stored reference pattern.
  • the depth camera 20 may be a time-of-flight depth camera.
  • the depth camera 20 includes an infrared projector and an infrared camera.
  • the infrared projector is used to emit uniform infrared light into the scene, and the infrared camera collects the infrared light reflected by the objects in the scene.
  • the processor 40 can calculate the depth information of the object in the scene based on the time difference between the emission time point of the infrared light and the reception time point of the infrared light.
  • the existing three-dimensional scene modeling method is generally to first collect multiple depth images and multiple visible light images of the scene, and then perform three-dimensional modeling of the scene based on the depth images and visible light images.
  • the complexity of the scene is high, for example, when there are many objects in the scene, or where the objects are placed in disorder, even if multiple depth images are taken, the objects in the scene may be partially blocked.
  • two pillows are stacked on the sofa, and one of the pillows is partially covered by the other pillow. At this time, neither the depth image nor the visible light image of the occluded part of the object can be collected, and then a complete 3D object model cannot be modeled during subsequent 3D modeling of the scene, which affects the integrity of the 3D modeling of the scene.
  • the processor 40 first controls the depth camera 20 to collect depth images of different shooting angles of the target modeling scene, and when the depth camera 20 is at a certain shooting angle to collect depth images, the processor 40 At the same time, the visible light camera 30 is controlled to correspondingly collect visible light images at the shooting angle. In this way, the processor 40 can obtain multiple depth images and multiple visible light images.
  • the multiple depth images correspond to the multiple visible light images one by one and the corresponding depth.
  • the image has the same or similar field of view as the visible image.
  • the processor 40 recognizes an occluded object in the scene based on the depth image and the visible light image, and a category of the occluded object.
  • the processor 40 calculates the estimated depth information and the estimated color information of the occluded part of the occluded object based on the actually measured measured depth information, the measured color information, and the type of the occluded object. It can be understood that after the category of the occluded object is identified, the shape of the occluded part of the occluded object can be roughly estimated according to the information of the category, and the measurement depth information can be used to estimate the size of the occluded part, and the measured color information can indicate the estimated occlusion Part of the color distribution and composition. Then, the depth information of the occluded part can be estimated based on the measured depth information, shape and estimated size information, and the estimated depth information can be obtained.
  • the occlusion can be estimated based on the measured color information, shape, and estimated color distribution and composition information. Part of the color information to get estimated color information. In this way, the depth information and color information of each occluded object in the scene can be supplemented and completed. Finally, the processor 40 can model a complete three-dimensional color model of the scene according to the measured depth information, measured color information, estimated depth information, and estimated color information.
  • the color information includes color information and black and white information.
  • the color information refers to colors such as red, yellow, blue, and green
  • the black and white information includes colors such as black, white, and gray.
  • the three-dimensional scene modeling method, the three-dimensional scene modeling device 10, and the electronic device 100 jointly estimate the occlusion based on three parameters of the measured scene depth information, color information, and the type of the occluded object identified.
  • the depth information and color information of the occluded part of the object so that the depth information and color information of the occluded part of the object in the scene are supplemented, which is conducive to improving the integrity of the 3D modeling of the scene.
  • the method for modeling a three-dimensional scene in the embodiment of the present application before step S5 further includes:
  • Step S41 includes:
  • S413 Perform stitching of depth images according to the unified depth information to obtain a wide-angle depth image.
  • the three-dimensional scene modeling apparatus 10 further includes a stitching module 14.
  • the splicing module 14 further includes a determining unit 141, a converting unit 142, and a splicing unit 143.
  • Step S41 may be implemented by the splicing module 14.
  • Step S411 may be implemented by the determination unit 141.
  • Step S412 may be implemented by the conversion unit 142.
  • Step S413 may be implemented by the splicing unit 143.
  • steps S41, S411, S412, and S413 may also be implemented by the processor 40. That is, the processor 40 can also be used to stitch multiple depth images to obtain a wide-angle depth image of the scene. Wherein, when the processor 40 executes step S41, the processor 40 actually executes determining the reference coordinate system, converting the measured depth information into the unified depth information in the reference coordinate system, and stitching the depth images according to the unified depth information to obtain Manipulation of wide-angle depth images.
  • the scenario shown in Figure 8 is the target construction scenario. Assuming that the field angles of the depth camera 20 and the visible light camera 30 are 90 degrees, the electronic device 100 needs to collect at least four depth images at different shooting angles and at least four visible light images at different shooting angles.
  • the user's station can be located in the center of the scene or around the scene. For example, the user can hold the electronic device 100 at point O (located near or near the center of the scene) of the scene, and shoot at least one of the positions facing A, B, C, and D, respectively.
  • the processor 40 After acquiring multiple depth images, the processor 40 needs to determine a reference coordinate system. For example, a user holds the electronic device 100 at point O, and acquires depth images A1, depth images B1, depth images C1, and depth images at the positions facing point A, facing B, facing C, and facing D, respectively.
  • the processor 40 first determines a reference coordinate system, for example, using point O of the scene as the origin (where , Point O can be obtained by processing visible light image recognition), construct a reference coordinate system xyz, where the xz plane can be used as a reference, and the point (x, y, z) of a point M in the scene (x, z) represents the point The projection position point of M on the xz plane, and y represents the vertical distance between the M point and the xz plane.
  • a reference coordinate system for example, using point O of the scene as the origin (where , Point O can be obtained by processing visible light image recognition), construct a reference coordinate system xyz, where the xz plane can be used as a reference, and the point (x, y, z) of a point M in the scene (x, z) represents the point The projection position point of M on the xz plane, and y represents the vertical distance between the M point and the xz plane.
  • the coordinates (x, y, z) can represent the position information of the M point in the scene relative to the origin O of the reference coordinate system, and also indicates the unified depth information of the M point in the reference coordinate system.
  • y can be understood Is the unified depth information of the point M in the reference coordinate system.
  • each depth image there is a pixel coordinate system uv (that is, the top left corner of the sensor array of the depth camera 20 is used as the origin, and the u axis and the v axis are parallel to the two vertical sides of the sensor array, respectively).
  • the pixel coordinates Is (u, v), and each pixel (u, v) corresponds to a certain area of an object in the scene.
  • Each pixel (u, v) corresponds to a measurement depth information.
  • Each measured depth information indicates the distance of the area of the object corresponding to the pixel (u, v) from the electronic device 100.
  • the processor 40 For capturing depth images at different viewing angles, the processor 40 needs to recalculate the rotation matrix and translation matrix between the reference coordinate system xyz and the pixel coordinate system uvd according to the distance relationship between the current position of the electronic device 100 and the point O, and then Based on the rotation matrix and translation matrix corresponding to the depth image taken at this perspective, (u, v, d) is converted into (x, y, z), so as to realize the conversion of measured depth information and unified depth information.
  • the processor 40 may stitch multiple depth images according to the unified depth information to obtain a wide-angle depth image.
  • multiple depth maps are located in a unified reference coordinate system, and one pixel point of each depth image corresponds to one coordinate (x, y, z), then Depth image stitching can be done through coordinate matching.
  • the coordinates of a pixel P1 in the depth image A1 are (x0, y0, z0), and the coordinates of a pixel P2 in the depth image A2 (x0, y0, z0).
  • the processor 40 can stitch the depth images by using the coordinate matching relationship and obtain a wide-angle depth image.
  • the wide-angle depth image indicates unified depth information of the entire scene.
  • the stitching of depth images based on the coordinate matching relationship requires the resolution of the depth image to be greater than a preset resolution. It can be understood that if the resolution of the depth image is low, the accuracy of the coordinates (x, y, z) will also be relatively low. At this time, matching directly based on the coordinates may cause that the points P1 and P2 do not actually overlap. , But the difference of an offset, and the value of the offset exceeds the error threshold. If the resolution of the image is high, the accuracy of the coordinates (x, y, z) will be relatively high. At this time, matching is performed based on the coordinates directly, even if the points P1 and P2 do not actually coincide, and there is an offset. Offset, but the value of offset will also be less than the error limit, that is, it is within the range allowed by the error, and will not affect the stitching of the depth image much.
  • the three-dimensional scene modeling method according to the embodiment of the present application before step S5 further includes:
  • S42 Stitch multiple visible light images to obtain a wide-angle visible light image of the scene.
  • step S42 may also be implemented by the splicing module 14. That is to say, the stitching module 14 is also used to stitch multiple visible light images to obtain a wide-angle visible light image of the scene.
  • step S42 may also be implemented by the processor 40. That is to say, the processor 40 can also be used to stitch multiple visible light images to obtain a wide-angle visible light image of the scene.
  • each depth image corresponds to a visible light image
  • the depth image A1 in the above example corresponds to visible light image A2
  • the depth image B1 corresponds to visible light image B2
  • the depth image C1 corresponds to visible light image C2
  • the depth image D1 corresponds to visible light image D2 .
  • the stitching of the visible light image is performed according to the matching information of the pixels in the depth image stitching process to match the pixels in the visible light image, thereby realizing the stitching of the visible light image.
  • the visible light image A2 and the visible light image B2 are spliced as an example. Because there is a certain relative position between the depth camera 20 and the visible light camera 30, that is, the field of view of the depth camera 20 and the field of view of the visible light camera 30 are not completely coincident. At this time, the processor 40 needs to align each group of the depth image and the visible light image with an associated relationship, and obtain an aligned corrected visible light image, and the pixels of the corrected visible light image correspond to the pixels of the depth image one to one. Subsequently, the processor 40 stitches the corrected visible light image A2 'and the corrected visible light image B2' based on the pixel correspondence between the depth image A1 and the depth image B1 to obtain a wide-angle visible light image. The wide-angle visible light image indicates the measured color information of the entire scene.
  • the processor 40 may also stitch the visible light image first, and then stitch the depth image based on the correspondence between the pixels in the visible light image.
  • the processor 40 may perform the stitching of the visible light image through feature matching. Specifically, first, the processor 40 first aligns each group of depth images and visible light images with an associated relationship. Subsequently, the processor 40 determines a registration parameter of the visible light image according to the mean structural similarity (Mean, Structural, Similarity Index, MSSIM).
  • the average structural similarity measures the similarity of the images from three aspects: brightness, contrast, and structure. Assuming that A2 and B2 are two visible light images to be judged for similarity, the similarity expressions in terms of brightness, contrast, and structure between them are as follows:
  • ⁇ A2 and ⁇ B2 represent the average of visible light image A2 and visible light image B2
  • ⁇ A2 and ⁇ B2 represent the variance of visible light image A2 and visible light image B2
  • ⁇ A2B2 represents the covariance of visible light image A2 and visible light image B2.
  • C 1 , C 2 and C 3 are constants.
  • SSIM (A2, B2) 1 (A2, B2) ⁇ c (A2, B2) ⁇ s (A2, B2).
  • the value range of SSIM is [0,1]. The larger the value, the smaller the distortion of the visible light image A2 and the visible light image B2, and the higher the similarity.
  • the processor 40 can stitch the visible light image A2 and the visible light image B2 based on the registration parameters, and obtain a wide-angle visible light image after stitching all visible light images. Further, the processor 40 may stitch the depth image A1 and the depth image B1 based on the pixel correspondence between the visible light image A2 and the visible light image B2, and obtain a wide-angle depth image after stitching all visible light images.
  • step S5 processes the depth image and the visible light image to identify occluded objects in the scene and the types of occluded objects include:
  • S51 Process a wide-angle depth image and a wide-angle visible light image to identify occluded objects and types of occluded objects in the scene.
  • Step S51 further includes:
  • S511 processing a wide-angle depth image and a wide-angle visible light image to extract occluded objects
  • S512 Find a two-dimensional object model corresponding to the occluded object from a two-dimensional object model library including a plurality of categories of two-dimensional object models.
  • the category of the two-dimensional object model is the category of the occluded object.
  • the finding subunit 1512 may be used to find a two-dimensional object model corresponding to the occluded object from a two-dimensional object model library including a plurality of categories of two-dimensional object models.
  • the category of the two-dimensional object model is the category of the occluded object.
  • steps S51, S511, and S512 may be implemented by the processor 40. That is to say, the processor 40 can also be used to process a wide-angle depth image and a wide-angle visible light image to identify occluded objects and types of occluded objects in the scene.
  • the processor 40 executes step S51, the processor 40 actually executes processing a wide-angle depth image and a wide-angle visible light image to extract an occluded object, and finds a correspondence to the occluded object from a two-dimensional object model library including a plurality of two-dimensional object models
  • the two-dimensional object model, the category of the two-dimensional object model is the operation of the category of occluding objects.
  • the two-dimensional object model library is stored in the storage element 50 of the electronic device 100.
  • the two-dimensional object model library has various types of two-dimensional object models, such as two-dimensional sofa models, two-dimensional vase models, two-dimensional square clock models, two-dimensional circular clock models, and the like.
  • the two-dimensional object model in the two-dimensional object model library can be obtained by collecting a large amount of two-dimensional image information in the early stage.
  • the category here is defined by the three types: type, shape, and color.
  • the two-dimensional object model may include a two-dimensional sofa model, a two-dimensional clock model, and the like.
  • the processor 40 first uses the edge detection algorithm to extract the contour information of the wide-angle visible light image, and optimizes the contour information of the visible light image based on the unified depth information indicated by the wide-angle depth image to make the contour information more accurate.
  • the processor 40 segments the visible light image based on the contour information to extract a plurality of objects. Further, the processor 40 determines whether the depth information of each object is complete based on the unified depth information. If there is a lack of depth information of an object, the object is considered to be occluded, that is, the object is an occluded object.
  • the processor 40 finds the two-dimensional object model corresponding to the occluded object in the two-dimensional object model library through feature matching, and uses the category of the found two-dimensional object model as the category of the occluded object. Show the category of the occluded object.
  • the processor 40 uses the information of the wide-angle depth image to optimize the contour information of the visible light image, and improves the accuracy of object extraction in the scene.
  • the processor 40 finds a two-dimensional object model corresponding to the occluded object from the two-dimensional object model library, which can improve the accuracy of the category recognition of the occluded object, and is beneficial to the subsequent calculation of the depth information and color information of the occluded part.
  • step S7 calculates the estimated depth information and estimated color information of the occluded object according to the measured depth information indicated by the depth image, the measured color information indicated by the visible light image, and the category, including:
  • S71 Obtain the size information of the occluded object according to the unified depth information and category;
  • S72 Find a three-dimensional object modeling method corresponding to the category of the occluded object in a three-dimensional object modeling method library including multiple three-dimensional object modeling methods.
  • the three-dimensional object modeling methods correspond to the two-dimensional object models one-to-one. ;
  • S73 Calculate the estimated depth information of the occluded object based on the size information, the coordinate information corresponding to the unified depth information, and the three-dimensional object modeling method;
  • S74 Calculate the estimated color information of the occluded object according to the measured color information of the occluded object and the two-dimensional object model corresponding to the occluded object.
  • the obtaining unit 171 may be configured to obtain the size information of the occluded object and the coordinate information relative to the reference coordinate system according to the unified depth information and category.
  • the finding unit 172 may be used to find a three-dimensional object modeling method corresponding to the category of the occluded object in a three-dimensional object modeling method library including a plurality of three-dimensional object modeling methods, a plurality of three-dimensional object modeling methods, and a plurality of two-dimensional object models.
  • the first calculation unit 173 may be configured to calculate the estimated depth information of the occluded object according to the size information, coordinate information corresponding to the unified depth information, and a three-dimensional object modeling method.
  • the second calculation unit 174 may be configured to calculate the estimated color information of the occluded object according to the measured color information of the occluded object and the two-dimensional object model corresponding to the occluded object.
  • the three-dimensional object modeling method library including a plurality of three-dimensional object modeling methods
  • multiple three-dimensional object modeling methods correspond one-to-one to multiple two-dimensional object models, and coordinate information and three-dimensional object modeling corresponding to size information and unified depth information
  • the method calculates the estimated depth information of the occluded object, and calculates the estimated color information of the occluded object according to the measured color information of the occluded object and the two-dimensional object model corresponding to the occluded object.
  • the three-dimensional object modeling method library is stored in the storage element 50 of the electronic device 100.
  • the three-dimensional object model library includes modeling methods for various types of objects.
  • the modeling methods for each type of objects are the modeling methods obtained by extracting the characteristics of objects in the real scene in the early stage and modeling based on the characteristics. .
  • each modeling method may correspond to objects of the same kind, same shape and same color, or each modeling method corresponds to objects of the same kind, same shape and different colors.
  • the processor 40 may determine the vertex position, shape, etc. of the occluded object according to the occluded object type. Further, based on the multiple occluded objects in the xyz reference coordinate system The coordinate information (x, y, z) and vertex position, shape and other information determine the size information of the occluded object. For example, the length, width, and height of the object are calculated based on the coordinate information corresponding to the vertex position. Subsequently, the processor 40 finds a three-dimensional object modeling method corresponding to the occluded object from the three-dimensional object modeling method library based on the category of the occluded object.
  • the depth information and color information of the occluded part of the occluded object can be supplemented, thereby obtaining complete depth information and color information of the entire scene.
  • step S9 to construct a three-dimensional color model of the scene according to the measured depth information, the measured color information, the estimated depth information, and the estimated color information includes:
  • the building module 19 includes a building unit 191 and a mapping unit 192.
  • Step S91 may be implemented by the construction unit 191.
  • Step S92 may be implemented by the mapping unit 192. That is, the construction unit 191 may be configured to construct a three-dimensional model of the scene according to the unified depth information and the estimated depth information.
  • the mapping unit 192 may be configured to map the three-dimensional model according to the measured color information and the estimated color information to obtain a three-dimensional color model.
  • steps S91 and S92 may also be implemented by the processor 40. That is, the processor 40 may be further configured to construct a three-dimensional model of the scene according to the unified depth information and the estimated depth information, and map the three-dimensional model according to the measured color information and the estimated color information to obtain a three-dimensional color model.
  • the processor 40 may perform three-dimensional modeling of the scene based on the depth information of the complete scene. Both the unified depth information and the estimated depth information use the reference coordinate system x-y-z as a reference. In fact, the coordinate information corresponding to the unified depth information and the estimated depth information can be understood as a point cloud required for three-dimensional modeling.
  • the processor 40 generates a plurality of triangular meshes based on the plurality of point clouds, where the shapes and areas of the plurality of triangles in the triangular mesh are similar.
  • a plurality of point clouds can be connected to form a three-dimensional model of the scene, and the triangular mesh formed after the point clouds are connected can simulate the surface of the three-dimensional model. Further, it is necessary to perform color rendering on the three-dimensional model.
  • the processor 40 may perform mapping processing on the three-dimensional model based on the measured color information and the estimated color information. Specifically, the color information corresponding to each triangular mesh on the visible light image is mapped onto the triangular mesh to implement the mapping of the three-dimensional model. , And finally get a three-dimensional color model of the scene.
  • the processor 40 may not perform the stitching process of the visible light image, but directly perform the extraction and recognition of the occlusion object based on the wide-angle depth image and the multiple visible light images
  • the calculation operations of estimated depth information and estimated color information are used to map the three-dimensional model directly based on the measured color information indicated by multiple visible light images and the calculated estimated color information in the subsequent mapping of the three-dimensional model to finally obtain the three-dimensional of the scene. Color model.
  • the processor 40 may also perform operations of extraction and recognition of occluded objects, calculation of estimated depth information, and estimated color information directly based on the depth image and the visible light image, and then perform subsequent operations based on the measured depth information and the estimated depth information.
  • the stitching of depth images, the stitching of visible images based on the measured color information and the estimated color information, and finally the construction of a three-dimensional model based on the wide-angle depth image, and the mapping of the three-dimensional model based on the wide-angle visible light image, finally the three-dimensional color model of the scene is obtained.
  • the present application further provides a computer device 200.
  • the computer device 200 includes a processor 210 and a memory 220.
  • Computer-readable instructions 230 are stored in the memory 220.
  • the processor 220 may execute the three-dimensional scene modeling method according to any one of the foregoing embodiments.
  • the processor 210 may perform the following steps:
  • the processor 210 may further perform the following steps:
  • the depth images are stitched according to the unified depth information to obtain a wide-angle depth image.
  • the application also provides a non-volatile computer-readable storage medium containing computer-executable instructions.
  • the processors are caused to execute the three-dimensional scene modeling method according to any one of the foregoing embodiments.
  • a three-dimensional color model of the scene is constructed based on the measured depth information, measured color information, estimated depth information, and estimated color information.
  • processors when executed by one or more processors, the processors can further perform the following steps:
  • the depth images are stitched according to the unified depth information to obtain a wide-angle depth image.
  • first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as “first” and “second” may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality” is at least two, for example, two, three, etc., unless it is specifically and specifically defined otherwise.
  • Any process or method description in a flowchart or otherwise described herein can be understood as a module, fragment, or portion of code that includes one or more executable instructions for implementing a particular logical function or step of a process
  • the scope of the preferred embodiments of this application includes additional implementations in which the functions may be performed out of the order shown or discussed, including performing the functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application pertain.
  • Logic and / or steps represented in a flowchart or otherwise described herein, for example, a sequenced list of executable instructions that may be considered to implement a logical function, may be embodied in any computer-readable medium, For use by, or in combination with, an instruction execution system, device, or device (such as a computer-based system, a system that includes a processor, or another system that can fetch and execute instructions from an instruction execution system, device, or device) Or equipment.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.
  • computer-readable media include the following: electrical connections (electronic devices) with one or more wirings, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disk read-only memory (CDROM).
  • the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable Processing to obtain the program electronically and then store it in computer memory.
  • each part of the application may be implemented by hardware, software, firmware, or a combination thereof.
  • multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it may be implemented using any one or a combination of the following techniques known in the art: Discrete logic circuits, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
  • a person of ordinary skill in the art can understand that all or part of the steps carried by the methods in the foregoing embodiments can be implemented by a program instructing related hardware.
  • the program can be stored in a computer-readable storage medium.
  • the program is When executed, one or a combination of the steps of the method embodiment is included.
  • each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist separately physically, or two or more units may be integrated into one module.
  • the above integrated modules may be implemented in the form of hardware or software functional modules. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
  • the aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

一种三维场景建模方法及装置（10）、电子装置（100）、可读存储介质和计算机设备（200）。三维场景建模方法包括：（S1）采集场景的深度图像；（S3）采集场景的可见光图像；（S5）处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别；（S7）根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息；（S9）根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。

Description

三维场景建模方法及装置、电子装置、可读存储介质及计算机设备
优先权信息
本申请请求2018年8月1日向中国国家知识产权局提交的、专利申请号为201810865825.8的专利申请的优先权和权益，并且通过参照将其全文并入此处。
技术领域
本申请涉及三维建模技术领域，特别涉及一种三维场景建模方法、三维场景建模装置、电子装置、非易失性计算机可读存储介质及计算机设备。
背景技术
现有的三维场景建模通常是通过深度相机拍摄场景的深度图像，并通过可见光相机拍摄二维可见光图像，结合深度图像的深度信息及二维可见光图像的色彩信息对场景进行三维建模。
发明内容
本申请的实施例提供了一种三维场景建模方法、三维场景建模装置、电子装置、非易失性计算机可读存储介质及计算机设备。
本申请实施方式的三维场景建模方法包括：采集所述场景的深度图像；采集所述场景的可见光图像；处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别；根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息；根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型。
本申请实施方式的三维场景建模装置包括第一采集模块、第二采集模块、处理模块、计算模块和构建模块。第一采集模块用于采集所述场景的深度图像。第二采集模块用于采集所述场景的可见光图像。处理模块用于处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别。计算模块用于根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息。构建模块用于根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型。
本申请实施方式的电子装置包括深度相机、可见光相机和处理器。所述深度相机用于采集所述场景的深度图像。所述可见光相机用于采集所述场景的可见光图像。所述处理器用于处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别，根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息，以及根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型。
本申请实施方式的包含计算可执行指令的非易失性计算机可读存储介质，当所述计算机可执行指令被一个或多个处理器执行时，使得所述处理器执行上述的三维场景建模方法。
本申请实施方式的计算机设备包括存储器及处理器，所述存储器中存储有计算机可读指令，所述指令被所述处理器执行时，使得所述处理器执行上述的三维场景建模方法。
本申请的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1是本申请某些实施方式的三维场景建模方法的流程示意图。
图2是本申请某些实施方式的三维场景建模装置的模块示意图。
图3是本申请某些实施方式的电子装置的结构示意图。
图4是本申请某些实施方式的三维场景建模方法的流程示意图。
图5是本申请某些实施方式的三维场景建模方法的流程示意图。
图6是本申请某些实施方式的三维场景建模装置的模块示意图。
图7是本申请某些实施方式的三维场景建模装置的拼接模块的模块示意图。
图8是本申请某些实施方式的三维场景建模方法的场景示意图。
图9是本申请某些实施方式的三维场景建模方法的流程示意图。
图10是本申请某些实施方式的三维场景建模方法的流程示意图。
图11是本申请某些实施方式的三维场景建模装置的模块示意图。
图12是本申请某些实施方式的三维场景建模装置的处理单元的模块示意图。
图13是本申请某些实施方式的三维场景建模方法的流程示意图。
图14是本申请某些实施方式的三维场景建模装置的计算模块的模块示意图。
图15是本申请某些实施方式的三维场景建模方法的流程示意图。
图16是本申请某些实施方式的三维场景建模装置的拼接模块的模块示意图。
图17是本申请某些实施方式的计算机设备的模块示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
本申请提供一种三维场景建模方法。三维场景建模方法包括：采集场景的深度图像；采集场景的可见光图像；处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别；根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息；根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
在某些实施方式中，深度图像包括多张，多张深度图像具有不同的拍摄角度，三维场景建模方法在处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别的步骤前还包括：拼接多张所述深度图像以得到所述场景的广角深度图像。
在某些实施方式中，拼接多张深度图像以得到场景的广角深度图像包括：确定参考坐标系；将测量深度信息转换为参考坐标系下的统一化深度信息；根据统一化深度信息做深度图像的拼接以得到广角深度图像。
在某些实施方式中，可见光图像包括多张，可见光图像具有不同的拍摄角度，多张可见光图像与多张深度图像一一对应。三维场景建模方法在处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别的步骤前还包括：拼接多张可见光图像以得到场景的广角可见光图像。
在某些实施方式中，处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别的步骤包括：处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别。
在某些实施方式中，处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别的步骤包括：处理广角深度图像及广角可见光图像以提取出遮挡物体；从包括多个类别的二维物体模型的二维物体模型库中寻找与遮挡物体对应的二维物体模型，二维物体模型的类别即为遮挡物体的类别。
在某些实施方式中，根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息的步骤包括：根据统一化深度信息及类别获取遮挡物体的尺寸信息；在包括多个三维物体建模方法的三维物体建模方法库中寻找与遮挡物体的类别对应的三维物体建模方法，多个三维物体建模方法与多个二维物体模型一一对应；根据尺寸信息、统一化深度信息对应 的坐标信息及三维物体建模方法计算遮挡物体的估计深度信息；根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型计算遮挡物体的估计色彩信息。
在某些实施方式中，根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型的步骤包括：根据统一化深度信息及估计深度信息构建场景的三维模型；根据测量色彩信息及估计色彩信息对三维模型进行贴图以得到三维色彩模型。
请参阅图2，本申请还提供一种三维场景建模装,10。三维场景建模装置10包括第一采集模块11、第二采集模块13、处理模块15、计算模块17及构建模块19。第一采集模块11用于采集场景的深度图像。第二采集模块13用于采集场景的可见光图像。处理模块15用于处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别。计算模块17用于根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息。构建模块19用于根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
请参阅图6，在某些实施方式中，深度图像包括多张，多张深度图像具有不同的拍摄角度。三维场景建模装置10还包括拼接模块14。拼接模块14用于拼接多张深度图像以得到场景的广角深度图像。
请参阅图7，在某些实施方式中，拼接模块包括确定单元141、转换单元142及拼接单元143。确定单元141用于确定参考坐标系。转换单元142用于将测量深度信息转换为参考坐标系下的统一化深度信息。拼接单元143用于根据统一化深度信息做深度图像的拼接以得到广角深度图像。
请再参阅图7，在某些实施方式中，可见光图像包括多张，多张可见光图像具有不同的拍摄角度，多张可见光图像与多张深度图像一一对应。拼接模块14还用于拼接多张可见光图像以得到场景的广角可见光图像。
请参阅图11，在某些实施方式中，处理模块15包括处理单元151。处理单元151用于处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别。
请参阅图12，在某些实施方式中，处理单元151包括处理子单元1511和寻找子单元1512。处理子单元1511用于处理广角深度图像及广角可见光图像以提取出遮挡物体。寻找子单元1512用于从包括多个类别的二维物体模型的二维物体模型库中寻找与遮挡物体对应的二维物体模型。二维物体模型的类别即为遮挡物体的类别。
请参阅图14，在某些实施方式中，计算模块包括获取单元171、寻找单元172、第一计算单元173及第二计算单元174。获取单元171用于根据统一化深度信息及类别获取遮挡物体的尺寸信息。寻找单元172用于在包括多个三维物体建模方法的三维物体建模方法库中寻找与遮挡物体的类别对应的三维物体建模方法，多个三维物体建模方法与多个二维物体模型一一对应。第一计算单元173用于根据尺寸信息、统一化深度信息对应的坐标信息及三维物体建模方法计算遮挡物体的估计深度信息。第二计算单元174用于根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型计算遮挡物体的估计色彩信息。
请参阅图16，在某些实施方式中，构建模块包括构建单元191及贴图单元192。构建单元191用于根据统一化深度信息及估计深度信息构建场景的三维模型。贴图单元192用于根据测量色彩信息及估计色彩信息对三维模型进行贴图以得到三维色彩模型。
请参阅图3，本申请还提供一种电子装置100。电子装置100包括深度相机20、可见光相机30和处理器40。深度相机20用于采集所述场景的深度图像。可见光相机30用于采集场景的可见光图像。处理器40用于：处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别，根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息，根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
请再参阅图3，在某些实施方式中，深度图像包括多张，多张深度图像具有不同的拍摄角度。处理器40还用于拼接多张深度图像以得到场景的广角深度图像。
请再参阅图3，在某些实施方式中，处理器40还用于：确定参考坐标系；将测量深度信息转换为参考坐标系下的统一化深度信息；根据统一化深度信息做深度图像的拼接以得到广角深度图像。
请再参阅图3，在某些实施方式中，可见光图像包括多张，多张可见光图像具有不同的拍摄角度， 多张可见光图像与多张深度图像一一对应。处理器40还用于拼接多张可见光图像以得到场景的广角可见光图像。
请再参阅图3，在某些实施方式中，处理器40还用于处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别。
请再参阅图3，在某些实施方式中，处理器40还用于：处理广角深度图像及广角可见光图像以提取出遮挡物体，从包括多个类别的二维物体模型的二维物体模型库中寻找与遮挡物体对应的二维物体模型，二维物体模型的类别即为遮挡物体的类别。
请再参阅图3，在某些实施方式中，处理器40还用于：根据统一化深度信息及类别获取遮挡物体的尺寸信息；在包括多个三维物体建模方法的三维物体建模方法库中寻找与遮挡物体的类别对应的三维物体建模方法，多个三维物体建模方法与多个二维物体模型一一对应；根据尺寸信息、统一化深度信息对应的坐标信息及三维物体建模方法计算遮挡物体的估计深度信息；根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型计算遮挡物体的估计色彩信息。
请再参阅图3，在某些实施方式中，处理器40还用于：根据统一化深度信息及估计深度信息构建场景的三维模型；根据测量色彩信息及估计色彩信息对三维模型进行贴图以得到三维色彩模型。
本申请还提供一种包含计算可执行指令的非易失性计算机可读存储介质。当所述计算机可执行指令被一个或多个处理器执行时，使得处理器执行上述任意一项实施方式所述的三维场景建模方法。
请参阅图17，本申请还提供一种计算机设备200.计算机设备200包括存储器220及处理器210。存储器220中存储有计算机可读指令230。指令被处理器210执行时，使得处理器210执行上述任意一项实施方式所述的三维场景建模方法。
请参阅图1，本申请提供一种三维场景建模方法。三维场景建模方法包括：
S1：采集场景的深度图像；
S3：采集场景的可见光图像；
S5：处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别；
S7：根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息；和
S9：根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
请参阅图2，本申请还提供一种三维场景建模装置10。本申请实施方式的三维场景建模方法可以由本申请实施方式的三维场景建模装置10实现。三维场景建模装置10包括第一采集模块11、第二采集模块13、处理模块15、计算模块17和构建模块19。步骤S1可以由第一采集模块11实现，步骤S3可以由第二采集模块13实现，步骤S5可以由处理模块15实现，步骤S7可以由计算模块17实现，步骤S9可以由构建模块19实现。
也即是说，第一采集模块11可用于采集场景的深度图像。第二采集模块13可用于采集场景的可见光图像。处理模块15可用于处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别。计算模块17可用于根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息。构建模块19可用于根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
请参阅图3，本申请还提供一种电子装置100。本申请实施方式的三维场景建模方法还可以由本申请实施方式的电子装置100实现。电子装置100包括深度相机20、可见光相机30和处理器40。步骤S1可以由深度相机20实现。步骤S3可以由可见光相机30实现，步骤S5、步骤S7和步骤S9均可以由处理器40实现。
也即是说，深度相机20可用于采集场景的深度图像。可见光相机30可用于采集场景的可见光图像。处理器40可用于处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别，根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息，以及根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模 型。
其中，电子装置100可以是智能手机、平板电脑、笔记本电脑、智能穿戴设备(如智能手表、智能手环、智能头盔、智能眼镜等)等。本申请实施方式以电子装置100是手机为例进行说明，可以理解，电子装置100的具体形式也可以是其他，在此不作限制。
深度相机20可以是可见光相机或红外相机。深度相机20为可见光相机时，该可见光相机与另一可见光相机30组成双目立体视觉系统，双目立体视觉系统基于三角测距法测量场景中物体的深度信息。深度相机20为红外相机时，该红外相机与可见光相机30组成双目立体视觉系统，双目立体视觉系统基于三角测距法测量场景中物体的深度信息。
深度相机20也可以是结构光深度相机。此时，深度相机20包括结构光投射器和红外相机。结构光投射器用于向场景中投射激光散斑，红外相机采集被场景中的物体调制后的激光图案。处理器40基于激光图案及预存的参考图案之间的偏移量即可计算出场景的深度信息。
深度相机20也可以是飞行时间深度相机。此时，深度相机20包括红外投射器和红外相机。红外投射器用于向场景中发射均匀的红外光，红外相机采集由场景中的物体反射回的红外光。处理器40基于红外光的发射时间点及红外光的接收时间点之间的时间差即可计算场景中物体的深度信息。
现有的三维场景建模的方式通常是先采集场景的多张深度图像和多张可见光图像，再基于深度图像和可见光图像对场景进行三维建模。在对场景进行三维建模时，通常需要对场景中的每一个物体做建模，从而得到多个三维物体模型，多个三维物体模型组成整个场景的三维模型。但在场景的复杂度较高时，例如，场景中的物体繁多，或者物体的放置位置杂乱无章时，即使拍摄了多张深度图像也会出现场景中的物体被部分遮挡的情况。如图8所示，沙发上堆叠放置有两个抱枕，其中一个抱枕的部分区域被另一个抱枕所遮挡。此时，物体被遮挡的部分的深度图像和可见光图像均没办法采集到，则后续做场景的三维建模时就无法建模出完整的三维物体模型，影响场景的三维建模的完整性。
本申请实施方式的三维场景建模方法，处理器40首先控制深度相机20采集目标建模场景的不同拍摄角度的深度图像，并且在深度相机20处于某一拍摄角度采集深度图像时，处理器40同时控制可见光相机30对应采集该拍摄角度下的可见光图像，如此，处理器40可获取到多张深度图像及多张可见光图像，多张深度图像与多张可见光图像一一对应，相对应的深度图像与可见光图像具有相同或相近的视场。随后，处理器40基于深度图像和可见光图像识别出场景中被遮挡的物体，以及该遮挡物体的类别。随后，处理器40再基于实际测得的测量深度信息、测量色彩信息和遮挡物体的类别来计算遮挡物体被遮挡的部分的估计深度信息和估计色彩信息。可以理解，识别出遮挡物体的类别后，可以根据类别的信息大致估计遮挡物体被遮挡部分的形状，而测量深度信息可以用于预估遮挡部分的尺寸的大小，测量色彩信息可以指示预估遮挡部分的色彩分布及构成。那么，根据测量深度信息、形状和预估的尺寸的信息即可预估遮挡部分的深度信息，得到估计深度信息，根据测量色彩信息、形状和预估的色彩分布及构成信息即可预估遮挡部分的色彩信息，得到估计色彩信息。如此，场景中每个遮挡物体被遮挡部分的深度信息和色彩信息均可以被补充完整。最后，处理器40根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息即可建模出场景的完整的三维色彩模型。
需要说明的是，上述色彩信息包括彩色信息及黑白信息。其中，彩色信息指的是红、黄、蓝、绿等色彩，黑白信息包括黑、白、灰等色彩。
综上，本申请实施方式的三维场景建模方法、三维场景建模装置10及电子装置100基于测量到的场景的深度信息、色彩信息以及识别到的遮挡物体的类别三个参数共同预估遮挡物体被遮挡部分的深度信息和色彩信息，从而使场景中遮挡物体被遮挡部分的深度信息和色彩信息得到补充，有利于提升场景的三维建模的完整性。
请一并参阅图4和图5，在某些实施方式中，本申请实施方式的三维场景建模方法在步骤S5前还包括：
S41：拼接多张深度图像以得到场景的广角深度图像。
其中，步骤S41包括：
S411：确定参考坐标系；
S412：将测量深度信息转换为参考坐标系下的统一化深度信息；和
S413：根据统一化深度信息做深度图像的拼接以得到广角深度图像。
请一并参阅图6和图7，在某些实施方式中，三维场景建模装置10还包括拼接模块14。其中，拼接模块14进一步包括确定单元141、转换单元142和拼接单元143。步骤S41可以由拼接模块14实现。步骤S411可以由确定单元141实现。步骤S412可以由转换单元142实现。步骤S413可以由拼接单元143实现。
也即是说，拼接模块14可用于拼接多张深度图像以得到场景的广角深度图像。确定模块可用于确定参考坐标系。转换单元142可用于将测量深度信息转换为参考坐标系下的统一化深度信息。拼接单元143可用于根据统一化深度信息做深度图像的拼接以得到广角深度图像。
请再参阅图3，在某些实施方式中，步骤S41、步骤S411、步骤S412和步骤S413还可以由处理器40实现。也即是说，处理器40还可用于拼接多张深度图像以得到场景的广角深度图像。其中，处理器40执行步骤S41时，处理器40实际上执行确定参考坐标系，将测量深度信息转换为参考坐标系下的统一化深度信息，以及根据统一化深度信息做深度图像的拼接以得到广角深度图像的操作。
具体地，由于受到深度相机20及可见光相机30的视场角的限制，要构建一个场景的三维色彩模型通常需要采集多张不同拍摄角度下的深度图像和可见光图像。如图8所示的场景为目标构建场景。假设深度相机20和可见光相机30的视场角均为90度，则电子装置100需要采集至少四张不同拍摄角度下的深度图像和至少四张不同拍摄角度下的可见光图像。其中，用户的站位可以是位于场景的中心，也可以位于场景的四周。例如，用户可以手持电子装置100站在场景的点O(位于场景中心位置或中心位置附近)处，在面向A点、面向B点、面向C点和面向D点的站位下分别拍摄至少一张深度图像和至少一张可见光图像；或者，用户可以手持电子装置100站在场景的四周的位置，即分别站在A点、B点、C点和D点的位置处，面向点O，分别拍摄至少一张深度图像和至少一张可见光图像。
在获取到多张深度图像后，处理器40需要确定一个参考坐标系。例如，用户手持电子装置100站在点O处，在面向A点、面向B点、面向C点和面向D点的站位下分别采集到深度图像A1、深度图像B1、深度图像C1和深度图像D1，并分别采集到对应各张深度图像的可见光图像A2、可见光图像B2、可见光图像C2和可见光图像D2，那么处理器40首先确定一个参考坐标系，例如，以场景的点O为原点(其中，点O可以通过处理可见光图像识别得到)，构建一个参考坐标系x-y-z，其中，可以以x-z平面为基准，场景中某一点M的坐标(x,y,z)中(x,z)表示点M在x-z面上的投影位置点，y表示M点与x-z平面的垂直距离。坐标(x,y,z)可以表示M点在场景中的相对于参考坐标系的原点O的位置信息，还指示了M点在参考坐标系下的统一化深度信息，具体地，y可理解为点M在参考坐标系下的统一化深度信息。
对于每一张深度图像，均具有一个像素坐标系u-v(即以深度相机20的传感器阵列的左上角顶点为原点，u轴和v轴分别平行于传感器阵列的两条垂直边)，像素的坐标为(u,v)，每一个像素(u,v)对应场景中的某一物体的某一区域。每一个像素(u,v)都对应有一个测量深度信息。每一个测量深度信息指示的是对应于像素(u,v)的物体的区域与电子装置100的距离。此时，可以在像素坐标系的基础上增加深度信息的内容，即像素坐标系变为u-v-d，每一个像素的坐标变为(u,v,d)，其中，d表示的是测量深度信息。随后，处理器40基于点O与电子装置100的位置关系，计算参考坐标系x-y-z与像素坐标系u-v-d之间的旋转矩阵和平移矩阵。随后，处理器40再根据旋转矩阵和平移矩阵将坐标(u,v,d)转换为(x,y,z)。处理器40通过上述的处理流程即可将测量深度信息d准换为统一化深度信息y。
对于不同视角下拍摄深度图像，处理器40需要根据电子装置100当前所处位置与点O之间的距离关系来重新计算参考坐标系x-y-z与像素坐标系u-v-d之间的旋转矩阵和平移矩阵，再基于与该视角下拍摄的深度图像对应的旋转矩阵和平移矩阵将(u,v,d)转换为(x,y,z)，从而实现测量深度信息与统一化深度信息的转换。
随后，处理器40可以根据统一化的深度信息将多张深度图像进行拼接以得到广角深度图像。具体 地，在将测量深度信息转换为统一化深度信息后，多张深度图位于一个统一的参考坐标系下，每一张深度图像的一个像素点对应一个坐标(x,y,z)，那么可以通过坐标匹配做深度图像的拼接。例如，在深度图像A1中某一个像素点P1的坐标为(x0,y0,z0)，在深度图像A2中某一个像素点P2的坐标(x0,y0,z0)，由于P1和P2在当前的参考坐标系下具有相同的坐标值，则说明像素点P1与像素点P2实际上为同一个点，深度图像A1和深度图像A2拼接时，像素点P1需要和像素点P2重合。如此，处理器40即可通过坐标的匹配关系进行深度图像的拼接，并得到广角深度图像，广角深度图像指示了整个场景的统一化深度信息。
需要说明的是，基于坐标的匹配关系进行深度图像的拼接要求深度图像的分辨率需要大于一个预设分辨率。可以理解，如果深度图像的分辨率较低，则坐标(x,y,z)的精确度也会相对较低，此时，直接根据坐标进行匹配，可能出现P1点和P2点实际上没有重合，而是相差一个偏移量offset，且offset的值超过误差界限值的问题。如果图像的分辨率较高，则坐标(x,y,z)的精确度也会相对较高，此时，直接根据坐标进行匹配，即使P1点和P2点实际上没有重合，相差一个偏移量offset，但offset的值也会小于误差界限值，即处于误差允许的范围内，不会对深度图像的拼接造成太大影响。
请再参阅图4，在拼接完深度图像后，在某些实施方式中，本申请实施方式的三维场景建模方法在步骤S5前还包括：
S42：拼接多张可见光图像以得到场景的广角可见光图像。
请再参阅图6，步骤S42也可以由拼接模块14实现。也即是说，拼接模块14还用于拼接多张可见光图像以得到场景的广角可见光图像。
请再参阅图3，步骤S42还可以由处理器40实现。也即是说，处理器40还可以用于拼接多张可见光图像以得到场景的广角可见光图像。
可以理解，由于每张深度图像对应一张可见光图像，例如上述例子中的深度图像A1对应可见光图像A2，深度图像B1对应可见光图像B2，深度图像C1对应可见光图像C2，深度图像D1对应可见光图像D2。那么，进一步地，以深度图像A1和深度图像B1的拼接为例，在深度图像A1和深度图像B1的拼接过程中，深度图像中的部分像素点P1与深度图像B2的部分像素点P2是重合的，则可见光图像的拼接依据深度图像拼接过程中像素点的匹配信息来进行可见光图像中像素点的匹配，从而实现可见光图像的拼接。
具体地，以拼接可见光图像A2和可见光图像B2为例。由于深度相机20与可见光相机30之间具有一定的相对位置，也即是说，深度相机20的视场与可见光相机30的视场不是完全重合的。此时，处理器40需要对每一组具有关联关系的深度图像和可见光图像做对齐，并得到对齐后的修正可见光图像，修正可见光图像的像素点与深度图像的像素点一一对应。随后，处理器40基于深度图像A1和深度图像B1的像素对应关系，对修正可见光图像A2’和修正可见光图像B2’做拼接，即可得到广角可见光图像。广角可见光图像指示了整个场景的测量色彩信息。
当然，在某些实施方式中，处理器40也可以先拼接可见光图像，再基于可见光图像中像素点的对应关系，进行深度图像的拼接。其中，处理器40可通过特征匹配来进行可见光图像的拼接。具体地，首先，处理器40先对每一组具有关联关系的深度图像和可见光图像做对齐。随后，处理器40根据平均结构相似度(Mean Structural Similarity Index,MSSIM)确定可见光图像的配准参数。平均结构相似度分别从亮度、对比度、结构这三个方面来衡量图像的相似性。假设A2和B2是要进行相似度评判的两幅可见光图像，则它们之间的亮度、对比度、结构这三个方面的相似性表达式分别如下：
Figure PCTCN2019088550-appb-000001
其中，μ A2和μ B2分别表示可见光图像A2和可见光图像B2的均值，σ A2和σ B2分别表示可见光图像A2和可见光图像B2的方差，σ A2B2表示可见光图像A2和可见光图像B2的协方差。C 1、C 2、C 3为常数。
平均结构相似度SSIM的表达式为：SSIM(A2,B2)＝l(A2,B2)×c(A2,B2)×s(A2,B2)。SSIM 的取值范围为[0,1]，取值越大，则可见光图像A2和可见光图像B2的失真越小，相似度越高。
在选定配准参数后，处理器40即可基于配准参数对可见光图像A2和可见光图像B2做拼接，在拼接完全部可见光图像后即可得到广角可见光图像。进一步地，处理器40可基于可见光图像A2和可见光图像B2的像素对应关系，拼接深度图像A1和深度图像B1，在拼接完全部的可见光图像后即可得到广角深度图像。
请一并参阅图9和图10，在某些实施方式中，在拼接完深度图像和可见光图像之后，步骤S5处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别包括：
S51：处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别。
其中，步骤S51进一步包括：
S511：处理广角深度图像及广角可见光图像以提取出遮挡物体；和
S512：从包括多个类别的二维物体模型的二维物体模型库中寻找与遮挡物体对应的二维物体模型，二维物体模型的类别即为遮挡物体的类别。
请一并参阅图11和图12，在某些实施方式中，处理模块15包括处理单元151。进一步地，处理单元151包括处理子单元1511和寻找子单元1512。步骤S51可以由处理单元151实现。步骤S511可以由处理子单元1511实现。步骤S512可以由寻找子单元1512实现。也即是说，处理单元151可用于处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别。处理子单元1511可用于处理广角深度图像及广角可见光图像以提取出遮挡物体。寻找子单元1512可用于从包括多个类别的二维物体模型的二维物体模型库中寻找与遮挡物体对应的二维物体模型，二维物体模型的类别即为遮挡物体的类别。
请再参阅图3，在某些实施方式中，步骤S51、步骤S511和步骤S512均可以由处理器40实现。也即是说，处理器40还可用于处理广角深度图像及广角可见光图像以识别场景中的遮挡物体及遮挡物体的类别。处理器40执行步骤S51时，处理器40实际上执行处理广角深度图像及广角可见光图像以提取出遮挡物体，从包括多个类别的二维物体模型的二维物体模型库中寻找与遮挡物体对应的二维物体模型，二维物体模型的类别即为遮挡物体的类别的操作。
其中，二维物体模型库存储在电子装置100的存储元件50中。二维物体模型库中具有多种类别的二维物体模型，例如，二维沙发模型，二维花瓶模型，二维方形时钟模型，二维圆形时钟模型等。二维物体模型库中的二维物体模型可以通过前期采集大量的二维图像信息得到。需要说明的是，此处的类别由种类、形状和色彩三者共同定义。举例来说，二维物体模型可以包括二维沙发模型、二维时钟模型等。其中，二维沙发模型和二维时钟模型的数量并不是唯一的，还会基于形状和颜色的改变进行对应的二维模型的存储。例如，二维时钟模型可以包括白色圆形时钟、蓝色圆形时钟、彩色方形时钟、绿色方形时钟等等。
具体地，处理器40首先利用边缘检测算法提取出广角可见光图像的轮廓信息，并基于广角深度图像指示的统一化深度信息对可见光图像的轮廓信息进行优化以使轮廓信息的精确度更高，随后，处理器40基于轮廓信息对可见光图像进行分割以提取出多个物体。进一步地，处理器40基于统一化深度信息判断每个物体的深度信息是否完整，若某一个物存在深度信息的缺失，则认为该物体受到遮挡，即该物体为遮挡物体。随后，处理器40通过特征匹配的方式在二维物体模型库中寻找与该遮挡物体对应的二维物体模型，并把找到的二维物体模型的类别作为遮挡物体的类别，如此，即可定出该遮挡物体的类别。
处理器40借助了广角深度图像的信息对可见光图像的轮廓信息进行优化，提升了场景中的物体提取的准确性。另外，处理器40从二维物体模型库中查找与遮挡物体对应的二维物体模型，可以提升遮挡物体的类别识别的准确性，有利于后续的遮挡部分的深度信息及色彩信息的计算。
请参阅图13，在某些实施方式中，步骤S7根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息包括：
S71：根据统一化深度信息及类别获取遮挡物体的尺寸信息；
S72：在包括多个三维物体建模方法的三维物体建模方法库中寻找与遮挡物体的类别对应的三维物 体建模方法，多个三维物体建模方法与多个二维物体模型一一对应；
S73：根据尺寸信息、统一化深度信息对应的坐标信息及三维物体建模方法计算遮挡物体的估计深度信息；和
S74：根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型计算遮挡物体的估计色彩信息。
请参阅图14，在某些实施方式中，计算模块17包括获取单元171、寻找单元172、第一计算单元173和第二计算单元174。步骤S71可以由获取单元171实现。步骤S72可以由寻找单元172实现。步骤S73可以由第一计算单元173实现。步骤S74可以由第二计算单元174实现。
也即是说，获取单元171可用于根据统一化深度信息及类别获取遮挡物体的尺寸信息及相对于参考坐标系的坐标信息。寻找单元172可用于在包括多个三维物体建模方法的三维物体建模方法库中寻找与遮挡物体的类别对应的三维物体建模方法，多个三维物体建模方法与多个二维物体模型一一对应。第一计算单元173可用于根据尺寸信息、统一化深度信息对应的坐标信息及三维物体建模方法计算遮挡物体的估计深度信息。第二计算单元174可用于根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型计算遮挡物体的估计色彩信息。
请再参阅图3，在某些实施方式中，步骤S71、步骤S72、步骤S73和步骤S74均可以由处理器40实现。也即是说，处理器40还可用于根据统一化深度信息及类别获取遮挡物体的尺寸信息及相对于参考坐标系的坐标信息，在包括多个三维物体建模方法的三维物体建模方法库中寻找与遮挡物体的类别对应的三维物体建模方法，多个三维物体建模方法与多个二维物体模型一一对应，根据尺寸信息、统一化深度信息对应的坐标信息及三维物体建模方法计算遮挡物体的估计深度信息，根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型计算遮挡物体的估计色彩信息。
其中，三维物体建模方法库存储在电子装置100的存储元件50中。三维物体模型库中包括多种类别的物体的建模方法，每种类别的物体的建模方法均是前期通过提取现实场景中物体的特征，并基于特征做建模后总结得到的建模方法。其中，每种建模方法可以对应同一种类、同一形状、同一颜色的物体，或者，每种建模方法对应同一种类、同一形状、不同颜色的物体。
具体地，在识别出遮挡物体及遮挡物体的类别后，处理器40可根据遮挡物体的类别确定遮挡物体的顶角位置、形状等，进一步地，基于x-y-z参考坐标系下的遮挡物体的多个坐标信息(x,y,z)及顶角位置、形状等信息确定出遮挡物体的尺寸信息，例如，基于顶角位置对应的坐标信息计算出物体的长度、宽度、高度等。随后，处理器40基于遮挡物体的类别从三维物体建模方法库中寻找与该遮挡物体对应的三维物体建模方法。随后，处理器40基于尺寸信息、统一化深度信息对应的坐标信息及三维物体建模方法对遮挡物体被遮挡部分的深度信息进行计算，得到估计深度信息。计算过程中，处理器40可以基于尺寸信息及坐标信息估计被遮挡部分的坐标信息，被遮挡部分的坐标信息中即包含了估计深度信息。另外，处理器40还可以根据遮挡物体的测量色彩信息及与遮挡物体对应的二维物体模型对遮挡物体被遮挡部分的色彩信息进行计算，得到估计色彩信息。可以理解，二维物体模型指示了物体的色彩信息，在遮挡物体与二维物体模型匹配后，可基于二维物体模型指示的色彩信息对遮挡物体被遮挡部分的色彩做估计。
如此，即可对遮挡物体被遮挡部分的深度信息和色彩信息做补充，从而得到整个场景的完整的深度信息和色彩信息。
请参阅图15，在某些实施方式中，步骤S9根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型包括：
S91：根据统一化深度信息及估计深度信息构建场景的三维模型；和
S92：根据测量色彩信息及估计色彩信息对三维模型进行贴图以得到三维色彩模型。
请参阅图16，在某些实施方式中，构建模块19包括构建单元191和贴图单元192。步骤S91可以由构建单元191实现。步骤S92可以由贴图单元192实现。也即是说，构建单元191可用于根据统一化深度信息及估计深度信息构建场景的三维模型。贴图单元192可用于根据测量色彩信息及估计色彩信息 对三维模型进行贴图以得到三维色彩模型。
请再参阅图3，在某些实施方式中，步骤S91和步骤S92还可以由处理器40实现。也即是说，处理器40还可用于根据统一化深度信息及估计深度信息构建场景的三维模型，以及根据测量色彩信息及估计色彩信息对三维模型进行贴图以得到三维色彩模型。
具体地，在获取到统一化深度信息和估计深度信息后，处理器40可基于完整的场景的深度信息做场景的三维建模。统一化深度信息和估计深度信息均以参考坐标系x-y-z作为参考，实际上，统一化深度信息和估计深度信息对应的多个坐标信息可以理解为三维建模所需的点云。处理器40基于多个点云生成多个三角网格，其中，三角网格中的多个三角形的形状和面积相近。如此，将多个点云进行连接即可形成场景的三维模型，点云连接后形成的三角网格可以模拟三维模型的表面。进一步地，还需要对三维模型进行色彩渲染。处理器40可基于测量色彩信息和估计色彩信息对三维模型做贴图处理，具体地，将可见光图像上与每个三角网格对应的色彩信息映射到三角网格上，即可实现三维模型的贴图，最终得到场景的三维色彩模型。
在某些实施方式中，处理器40在拼接多张深度图像得到广角深度图像后，可以不做可见光图像的拼接处理，而是直接基于广角深度图像、多张可见光图像执行遮挡物体的提取和识别、估计深度信息和估计色彩信息的计算的操作，在后续对三维模型贴图时，直接基于多张可见光图像指示的测量色彩信息及计算得到的估计色彩信息对三维模型做贴图，最终得到场景的三维色彩模型。
在某些实施方式中，处理器40也可直接基于深度图像和可见光图像执行遮挡物体的提取和识别、估计深度信息和估计色彩信息的计算的操作，后续再基于测量深度信息和估计深度信息做深度图像的拼接、基于测量色彩信息和估计色彩信息做可见光图像的拼接，最后基于广角深度图像做三维模型的构建，基于广角可见光图像做三维模型的贴图，最终得到场景的三维色彩模型。
在某些实施方式中，处理器40也可直接基于深度图像和可见光图像执行遮挡物体的提取和识别、估计深度信息和估计色彩信息的计算的操作，后续再基于测量深度信息和估计深度信息做深度图像的拼接得到广角深度图像，最后基于广角深度图像做三维模型的构建，基于多张可见光图像做三维模型的贴图，最终得到场景的三维色彩模型。
请参阅图17，本申请还提供一种计算机设备200。计算机设备200包括处理器210和存储器220。存储器中220存储有计算机可读指令230。当计算机可读指令230被处理器220执行时，处理器220可以执行上述任意一项实施方式所述的三维场景建模方法。
计算机设备200可以是上述任意一项实施方式所述的电子装置100。此时，处理器210可为处理器40，存储器220可为存储元件50。计算机可读指令230可存储在存储元件50中。
例如，当指令被处理器210执行时，处理器210可以执行以下步骤：
控制深度相机20采集场景的深度图像；
控制可见光相机30采集场景的可见光图像；
处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别；
根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息；和
根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
再例如，当指令被处理器210执行时，处理器210还可以执行以下步骤：
确定参考坐标系；
将测量深度信息转换为参考坐标系下的统一化深度信息；和
根据统一化深度信息做深度图像的拼接以得到广角深度图像。
本申请还提供一种包含计算机可执行指令的非易失性计算机可读存储介质。当计算机可执行指令被一个或多个处理器执行时，使得处理器执行上述任意一项实施方式所述的三维场景建模方法。
例如，当计算机可执行指令被一个或多个处理器执行时，使得处理器可以执行以下步骤：
控制深度相机20采集场景的深度图像；
控制可见光相机30采集场景的可见光图像；
处理深度图像及可见光图像以识别场景中的遮挡物体及遮挡物体的类别；
根据深度图像指示的测量深度信息、可见光图像指示的测量色彩信息及类别计算遮挡物体的估计深度信息和估计色彩信息；和
根据测量深度信息、测量色彩信息、估计深度信息和估计色彩信息构建场景的三维色彩模型。
再例如，当计算机可执行指令被一个或多个处理器执行时，使得处理器还可以执行以下步骤：
确定参考坐标系；
将测量深度信息转换为参考坐标系下的统一化深度信息；和
根据统一化深度信息做深度图像的拼接以得到广角深度图像。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产 品销售或使用时，也可以存储在一个计算机可读取存储介质中。
上述提到的存储介质可以是只读存储器，磁盘或光盘等。尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (26)

  1. 一种三维场景建模方法，其特征在于，所述三维场景建模方法包括：
    采集所述场景的深度图像；
    采集所述场景的可见光图像；
    处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别；
    根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息；和
    根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型。
  2. 根据权利要求1所述的三维场景建模方法，其特征在于，所述深度图像包括多张，多张所述深度图像具有不同的拍摄角度，所述三维场景建模方法在所述处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别的步骤前还包括：
    拼接多张所述深度图像以得到所述场景的广角深度图像。
  3. 根据权利要求2所述的三维场景建模方法，其特征在于，所述拼接多张所述深度图像以得到所述场景的广角深度图像包括：
    确定参考坐标系；
    将所述测量深度信息转换为所述参考坐标系下的统一化深度信息；和
    根据所述统一化深度信息做所述深度图像的拼接以得到所述广角深度图像。
  4. 根据权利要求2所述的三维场景建模方法，其特征在于，所述可见光图像包括多张，多张所述可见光图像具有不同的拍摄角度，多张所述可见光图像与多张所述深度图像一一对应，所述三维场景建模方法在所述处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别的步骤前还包括：
    拼接多张所述可见光图像以得到所述场景的广角可见光图像。
  5. 根据权利要求4所述的三维场景建模方法，其特征在于，所述处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别的步骤包括：
    处理所述广角深度图像及所述广角可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别。
  6. 根据权利要求5所述的三维场景建模方法，其特征在于，所述处理所述广角深度图像及所述广角可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别的步骤包括：
    处理所述广角深度图像及所述广角可见光图像以提取出所述遮挡物体；和
    从包括多个类别的二维物体模型的二维物体模型库中寻找与所述遮挡物体对应的二维物体模型，所述二维物体模型的类别即为所述遮挡物体的类别。
  7. 根据权利要求6所述的三维场景建模方法，其特征在于，所述根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息的步骤包括：
    根据所述统一化深度信息及所述类别获取所述遮挡物体的尺寸信息；
    在包括多个三维物体建模方法的三维物体建模方法库中寻找与所述遮挡物体的类别对应的所述三维物体建模方法，多个所述三维物体建模方法与多个所述二维物体模型一一对应；
    根据所述尺寸信息、所述统一化深度信息对应的坐标信息及所述三维物体建模方法计算所述遮挡物体的估计深度信息；和
    根据所述遮挡物体的测量色彩信息及与所述遮挡物体对应的二维物体模型计算所述遮挡物体的估计色彩信息。
  8. 根据权利要求3所述的三维场景建模方法，其特征在于，所述根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型的步骤包括：
    根据所述统一化深度信息及所述估计深度信息构建所述场景的三维模型；和
    根据所述测量色彩信息及所述估计色彩信息对所述三维模型进行贴图以得到所述三维色彩模型。
  9. 一种三维场景建模装置，其特征在于，所述三维场景建模装置包括：
    第一采集模块，用于采集所述场景的深度图像；
    第二采集模块，用于采集所述场景的可见光图像；
    处理模块，用于处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别；
    计算模块，用于根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息；和
    构建模块，用于根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型。
  10. 根据权利要求9所述的三维场景建模装置，其特征在于，所述深度图像包括多张，多张所述深度图像具有不同的拍摄角度，所述三维场景建模装置还包括拼接模块，所述拼接模块用于拼接多张所述深度图像以得到所述场景的广角深度图像。
  11. 根据权利要求10所述的三维场景建模装置，其特征在于，所述拼接模块包括：
    确定单元，用于确定参考坐标系；
    转换单元，用于将所述测量深度信息转换为所述参考坐标系下的统一化深度信息；和
    拼接单元，用于根据所述统一化深度信息做所述深度图像的拼接以得到所述广角深度图像。
  12. 根据权利要求10所述的三维场景建模装置，其特征在于，所述可见光图像包括多张，多张所述可见光图像具有不同的拍摄角度，多张所述可见光图像与多张所述深度图像一一对应，所述拼接模块还用于拼接多张所述可见光图像以得到所述场景的广角可见光图像。
  13. 根据权利要求12所述的三维场景建模装置，其特征在于，所述处理模块包括处理单元，所述处理单元用于处理所述广角深度图像及所述广角可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别。
  14. 根据权利要求13所述的三维场景建模装置，其特征在于，所述处理单元包括：
    处理子单元，用于处理所述广角深度图像及所述广角可见光图像以提取出所述遮挡物体；和
    寻找子单元，用于从包括多个类别的二维物体模型的二维物体模型库中寻找与所述遮挡物体对应的二维物体模型，所述二维物体模型的类别即为所述遮挡物体的类别。
  15. 根据权利要求14所述的三维场景建模装置，其特征在于，所述计算模块包括：
    获取单元，用于根据所述统一化深度信息及所述类别获取所述遮挡物体的尺寸信息；
    寻找单元，用于在包括多个三维物体建模方法的三维物体建模方法库中寻找与所述遮挡物体的类别对应的所述三维物体建模方法，多个所述三维物体建模方法与多个所述二维物体模型一一对应；
    第一计算单元，用于根据所述尺寸信息、所述统一化深度信息对应的坐标信息及所述三维物体建模方法计算所述遮挡物体的估计深度信息；和
    第二计算单元，用于根据所述遮挡物体的测量色彩信息及与所述遮挡物体对应的二维物体模型计算所述遮挡物体的估计色彩信息。
  16. 根据权利要求11所述的三维场景建模装置，其特征在于，所述构建模块包括：
    构建单元，用于根据所述统一化深度信息及所述估计深度信息构建所述场景的三维模型；和
    贴图单元，用于根据所述测量色彩信息及所述估计色彩信息对所述三维模型进行贴图以得到所述三维色彩模型。
  17. 一种电子装置，其特征在于，所述电子装置包括：
    深度相机，所述深度相机用于采集所述场景的深度图像；
    可见光相机，所述可见光相机用于采集所述场景的可见光图像；和
    处理器，所述处理器用于：
    处理所述深度图像及所述可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别；
    根据所述深度图像指示的测量深度信息、所述可见光图像指示的测量色彩信息及所述类别计算所述遮挡物体的估计深度信息和估计色彩信息；和
    根据所述测量深度信息、所述测量色彩信息、所述估计深度信息和所述估计色彩信息构建所述场景的三维色彩模型。
  18. 根据权利要求17所述的电子装置，其特征在于，所述深度图像包括多张，多张所述深度图像具有不同的拍摄角度，所述处理器还用于：
    拼接多张所述深度图像以得到所述场景的广角深度图像。
  19. 根据权利要求18所述的电子装置，其特征在于，所述处理器还用于：
    确定参考坐标系；
    将所述测量深度信息转换为所述参考坐标系下的统一化深度信息；和
    根据所述统一化深度信息做所述深度图像的拼接以得到所述广角深度图像。
  20. 根据权利要求18所述的电子装置，其特征在于，所述可见光图像包括多张，多张所述可见光图像具有不同的拍摄角度，多张所述可见光图像与多张所述深度图像一一对应，所述处理器还用于：
    拼接多张所述可见光图像以得到所述场景的广角可见光图像。
  21. 根据权利要求20所述的电子装置，其特征在于，所述处理器还用于：
    处理所述广角深度图像及所述广角可见光图像以识别所述场景中的遮挡物体及所述遮挡物体的类别。
  22. 根据权利要求21所述的电子装置，其特征在于，所述处理器还用于：
    处理所述广角深度图像及所述广角可见光图像以提取出所述遮挡物体；和
    从包括多个类别的二维物体模型的二维物体模型库中寻找与所述遮挡物体对应的二维物体模型，所述二维物体模型的类别即为所述遮挡物体的类别。
  23. 根据权利要求22所述的电子装置，其特征在于，所述处理器还用于：
    根据所述统一化深度信息及所述类别获取所述遮挡物体的尺寸信息；
    在包括多个三维物体建模方法的三维物体建模方法库中寻找与所述遮挡物体的类别对应的所述三维物体建模方法，多个所述三维物体建模方法与多个所述二维物体模型一一对应；
    根据所述尺寸信息、所述统一化深度信息对应的坐标信息及所述三维物体建模方法计算所述遮挡物体的估计深度信息；和
    根据所述遮挡物体的测量色彩信息及与所述遮挡物体对应的二维物体模型计算所述遮挡物体的估计色彩信息。
  24. 根据权利要求19所述的电子装置，其特征在于，所述处理器还用于：
    根据所述统一化深度信息及所述估计深度信息构建所述场景的三维模型；和
    根据所述测量色彩信息及所述估计色彩信息对所述三维模型进行贴图以得到所述三维色彩模型。
  25. 一种包含计算可执行指令的非易失性计算机可读存储介质，其特征在于，当所述计算机可执行指令被一个或多个处理器执行时，使得所述处理器执行权利要求1至8任意一项所述的三维场景建模方法。
  26. 一种计算机设备，其特征在于，包括存储器及处理器，所述存储器中存储有计算机可读指令，所述指令被所述处理器执行时，使得所述处理器执行权利要求1至8任意一项所述的三维场景建模方法。
PCT/CN2019/088550 2018-08-01 2019-05-27 三维场景建模方法及装置、电子装置、可读存储介质及计算机设备 WO2020024684A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810865825.8 2018-08-01
CN201810865825.8A CN110798677B (zh) 2018-08-01 2018-08-01 三维场景建模方法及装置、电子装置、可读存储介质及计算机设备

Publications (1)

Publication Number Publication Date
WO2020024684A1 true WO2020024684A1 (zh) 2020-02-06

Family

ID=69232145

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/088550 WO2020024684A1 (zh) 2018-08-01 2019-05-27 三维场景建模方法及装置、电子装置、可读存储介质及计算机设备

Country Status (2)

Country Link
CN (1) CN110798677B (zh)
WO (1) WO2020024684A1 (zh)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111627061B (zh) * 2020-06-03 2023-07-11 如你所视(北京)科技有限公司 位姿检测方法、装置以及电子设备、存储介质
CN112383746A (zh) * 2020-10-29 2021-02-19 北京软通智慧城市科技有限公司 一种三维场景中的视频监控方法、装置、电子设备和存储介质
CN112367514B (zh) * 2020-10-30 2022-12-09 京东方科技集团股份有限公司 三维场景构建方法、装置、系统和存储介质
CN112561989B (zh) * 2020-12-23 2022-11-01 清华大学 一种建造场景下吊装对象的识别方法
CN114049444B (zh) * 2022-01-13 2022-04-15 深圳市其域创新科技有限公司 一种3d场景生成方法及装置
CN116071724B (zh) * 2023-03-03 2023-08-04 安徽蔚来智驾科技有限公司 车载相机遮挡场景识别方法、电子设备、存储介质及车辆

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101271526A (zh) * 2008-04-22 2008-09-24 深圳先进技术研究院 一种图像处理中物体自动识别并三维重建的方法
CN105825544A (zh) * 2015-11-25 2016-08-03 维沃移动通信有限公司 一种图像处理方法及移动终端
US9661310B2 (en) * 2011-11-28 2017-05-23 ArcSoft Hanzhou Co., Ltd. Image depth recovering method and stereo image fetching device thereof
CN107172408A (zh) * 2017-05-18 2017-09-15 温州大学 单反摄影技术结合传统测量的高精度古建筑数字建模方法

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100573190C (zh) * 2007-09-04 2009-12-23 西安电子科技大学 基于支撑矢量机的合成孔径雷达遮挡目标识别方法
JP5697989B2 (ja) * 2007-12-26 2015-04-08 コーニンクレッカ フィリップス エヌ ヴェ グラフィックスオブジェクトを重ね合わせるための画像プロセッサ
US8866920B2 (en) * 2008-05-20 2014-10-21 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
EP2451164B1 (en) * 2010-11-08 2017-05-03 Telefonaktiebolaget LM Ericsson (publ) Improved view synthesis
KR20120049636A (ko) * 2010-11-09 2012-05-17 삼성전자주식회사 영상 처리 장치 및 방법
CN102622769B (zh) * 2012-03-19 2015-03-04 厦门大学 一种在动态场景下以深度为主导线索的多目标跟踪方法
CN104662589B (zh) * 2012-08-21 2017-08-04 派力肯影像公司 用于使用阵列照相机捕捉的图像中的视差检测和校正的系统和方法
CN105205866B (zh) * 2015-08-30 2018-04-13 浙江中测新图地理信息技术有限公司 基于密集点云的城市三维模型快速构建方法
CN106503170B (zh) * 2016-10-31 2019-09-27 清华大学 一种基于遮挡维度的图像库构建方法
CN106651943B (zh) * 2016-12-30 2019-07-23 杭州电子科技大学 一种基于遮挡几何互补模型的光场相机深度估计方法
CN107145867A (zh) * 2017-05-09 2017-09-08 电子科技大学 基于多任务深度学习的人脸及人脸遮挡物检测方法
CN107341827B (zh) * 2017-07-27 2023-01-24 腾讯科技（深圳）有限公司 一种视频处理方法、装置和存储介质
CN107360354B (zh) * 2017-07-31 2020-06-26 Oppo广东移动通信有限公司 拍照方法、装置、移动终端和计算机可读存储介质
CN108121972A (zh) * 2017-12-25 2018-06-05 北京航空航天大学 一种局部遮挡条件下的目标识别方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101271526A (zh) * 2008-04-22 2008-09-24 深圳先进技术研究院 一种图像处理中物体自动识别并三维重建的方法
US9661310B2 (en) * 2011-11-28 2017-05-23 ArcSoft Hanzhou Co., Ltd. Image depth recovering method and stereo image fetching device thereof
CN105825544A (zh) * 2015-11-25 2016-08-03 维沃移动通信有限公司 一种图像处理方法及移动终端
CN107172408A (zh) * 2017-05-18 2017-09-15 温州大学 单反摄影技术结合传统测量的高精度古建筑数字建模方法

Also Published As

Publication number Publication date
CN110798677B (zh) 2021-08-31
CN110798677A (zh) 2020-02-14

Similar Documents

Publication Publication Date Title
WO2020024684A1 (zh) 三维场景建模方法及装置、电子装置、可读存储介质及计算机设备
CN110568447B (zh) 视觉定位的方法、装置及计算机可读介质
US10311648B2 (en) Systems and methods for scanning three-dimensional objects
US9129435B2 (en) Method for creating 3-D models by stitching multiple partial 3-D models
CN109801374B (zh) 一种通过多角度图像集重构三维模型的方法、介质及系统
US8452081B2 (en) Forming 3D models using multiple images
US10846844B1 (en) Collaborative disparity decomposition
US8447099B2 (en) Forming 3D models using two images
TW201915944A (zh) 圖像處理方法、裝置、系統和儲存介質
CN109211207B (zh) 一种基于机器视觉的螺钉识别与定位装置
WO2021203883A1 (zh) 三维扫描方法、三维扫描系统和计算机可读存储介质
WO2020038386A1 (zh) 确定单目视觉重建中的尺度因子
WO2021136386A1 (zh) 数据处理方法、终端和服务器
CN110567441B (zh) 基于粒子滤波的定位方法、定位装置、建图及定位的方法
WO2022078488A1 (zh) 定位方法、装置、自移动设备和存储介质
US20210374978A1 (en) Capturing environmental scans using anchor objects for registration
CN108629829A (zh) 一种球幕相机与深度相机结合的三维建模方法和系统
WO2021238070A1 (zh) 三维图像的生成方法、装置及计算机设备
TWI599987B (zh) 點雲拼接系統及方法
WO2020224199A1 (zh) 鱼眼相机标定系统、方法、装置、电子设备及存储介质
CN108876891B (zh) 人脸图像数据采集方法及人脸图像数据采集装置
TWI595446B (zh) 擴充實境中基於深度攝影機之遮蔽邊緣品質改善方法
US11551368B2 (en) Electronic devices, methods, and computer program products for controlling 3D modeling operations based on pose metrics
WO2022011560A1 (zh) 图像裁剪方法与装置、电子设备及存储介质
US11516448B2 (en) Method and apparatus for compensating projection images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19844592

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19844592

Country of ref document: EP

Kind code of ref document: A1