WO2017020467A1 - 场景重建方法、装置、终端设备及存储介质 - Google Patents

场景重建方法、装置、终端设备及存储介质 Download PDF

Info

Publication number
WO2017020467A1
WO2017020467A1 PCT/CN2015/096623 CN2015096623W WO2017020467A1 WO 2017020467 A1 WO2017020467 A1 WO 2017020467A1 CN 2015096623 W CN2015096623 W CN 2015096623W WO 2017020467 A1 WO2017020467 A1 WO 2017020467A1
Authority
WO
WIPO (PCT)
Prior art keywords
scene
picture
reconstructed
feature area
feature
Prior art date
Application number
PCT/CN2015/096623
Other languages
English (en)
French (fr)
Inventor
杜堂武
艾锐
蒋昭炎
刘丽
朗咸朋
Original Assignee
百度在线网络技术（北京）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 百度在线网络技术（北京）有限公司 filed Critical 百度在线网络技术（北京）有限公司
Priority to JP2018506162A priority Critical patent/JP6553285B2/ja
Priority to US15/749,642 priority patent/US10467800B2/en
Priority to EP15900236.9A priority patent/EP3324368B1/en
Priority to KR1020187005930A priority patent/KR102033262B1/ko
Publication of WO2017020467A1 publication Critical patent/WO2017020467A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/80—2D [Two Dimensional] animation, e.g. using sprites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/416—Exact reconstruction

Definitions

  • the embodiments of the present invention relate to the field of image processing technologies, and in particular, to a scene reconstruction method, apparatus, terminal device, and storage medium.
  • a powerful search engine or a specific collection method can be used to collect a large number of images taken by many users. These images will be distributed at different times of the day and distributed in different seasons of the year. It will even be distributed in different years. Based on this, through the image screening and reconstruction, the public can appreciate the different styles of these monuments from a large time scale and spatial scale.
  • the embodiment of the invention provides a scene reconstruction method, device, terminal device and storage medium, which can improve reconstruction efficiency.
  • an embodiment of the present invention provides a scenario reconstruction method, including:
  • the three-dimensional reconstruction of the to-be-reconstructed scene is performed according to the scene feature area in the picture, and the scene to be reconstructed is generated and rendered.
  • the embodiment of the present invention further provides a scenario reconstruction apparatus, including:
  • a picture obtaining module configured to acquire a first picture set that matches a scene to be reconstructed
  • a feature extraction module configured to extract, by using a feature extraction algorithm, at least one feature region of the image in the first image set
  • a feature recognition module configured to identify the feature area to obtain a scene feature area in the picture
  • a reconstruction module configured to perform three-dimensional reconstruction on the to-be-reconstructed scene according to the feature feature area in the picture, and generate and generate the to-be-reconstructed scene.
  • the embodiment of the present invention further provides a terminal device that implements scene reconstruction, including:
  • One or more processors are One or more processors;
  • One or more modules the one or more modules being stored in the memory, and when executed by the one or more processors, performing the following operations:
  • the three-dimensional reconstruction of the to-be-reconstructed scene is performed according to the scene feature area in the picture, and the scene to be reconstructed is generated and rendered.
  • an embodiment of the present invention further provides a non-volatile computer storage medium, where the computer storage medium stores one or more modules, when the one or more modules are executed by a device that performs a scene reconstruction method.
  • the device is caused to perform the following operations:
  • the three-dimensional reconstruction of the to-be-reconstructed scene is performed according to the scene feature area in the picture, and the scene to be reconstructed is generated and rendered.
  • the technical solution of the embodiment of the present invention filters out some ineffective and unstable feature regions, and performs three-dimensional reconstruction on the to-be-reconstructed scene only according to the scene feature region associated with the to-be-reconstructed scene, thereby improving reconstruction efficiency and accuracy. Sex.
  • FIG. 1 is a schematic flowchart of a scenario reconstruction method according to Embodiment 1 of the present invention
  • FIG. 2 is a schematic structural diagram of a scene reconstruction apparatus according to Embodiment 2 of the present invention.
  • FIG. 3 is a schematic structural diagram of a terminal device for implementing scene reconstruction according to Embodiment 3 of the present invention.
  • the executor of the scenario reconstruction method provided by the embodiment of the present invention may be the scenario reconstruction device provided by the embodiment of the present invention or the server device integrated with the scenario reconstruction device, and the scenario reconstruction device may be implemented by using hardware or software.
  • FIG. 1 is a schematic flowchart of a method for reconstructing a scenario according to Embodiment 1 of the present invention. As shown in FIG.
  • the scene to be reconstructed may be some tourist attractions, historical monuments, buildings, and the like.
  • the first picture set includes at least one picture that matches the scene to be reconstructed. Specifically, it may be obtained by inputting a keyword search related to the scene to be reconstructed on the Internet, or may be obtained from a User Generated Content (UGC) image library.
  • UPC User Generated Content
  • the UGC image library stores the image content shared by the user on the Internet, and most of the image content is derived from photos taken by the user, or may be pictures created by the user. Due to the diversity of cameras on the market, and due to different shooting time, location and mode, the UGC photo library records the same scene at different viewpoints, time (four seasons change or morning and evening alternate), lighting conditions (yin, sunny, rain or Different appearances under the snow, with the rapid increase in the amount of data uploaded by users, the UGC image library has the characteristics of wide coverage and low data acquisition cost. Therefore, a more valuable graph can be obtained from the UGC image library. Slice content.
  • the feature extraction algorithm may include a corner detection algorithm and a local invariant feature point extraction method.
  • the pre-processing of the pictures is required, and the main processing is picture feature segmentation.
  • the image is divided into a number of specific areas with unique properties, and the target object (for example, the character area, the scenery area, and the scene area in the picture) is proposed.
  • the image segmentation methods that can be employed mainly include a threshold-based segmentation method, a region-based segmentation method, an edge-based segmentation method, and a specific theory-based segmentation method.
  • the feature area in the picture After the feature area in the picture is extracted, the feature area needs to be identified, so as to distinguish the feature area, and the feature area in the picture that matches the scene to be reconstructed, that is, the scene feature area is determined.
  • the feature area of the picture can be identified by the following feature recognition algorithm: image recognition algorithm based on graph statistical features, object recognition algorithm based on HOG feature, and object recognition algorithm based on local feature.
  • S14 Perform three-dimensional reconstruction on the to-be-reconstructed scene according to the scene feature area in the picture, and render the scene to be reconstructed.
  • the scene feature area may be an outline of a scene that matches an attraction in the scene to be reconstructed. Taking the scenic spot in the scene to be reconstructed as the Longmen Grottoes as an example, the feature area can be selected from the outlines of a plurality of objects of the Buddha image in the Longmen Grottoes, such as the face, eyes, and hands of the Buddha image.
  • the existing three-dimensional reconstruction process can adopt some existing reconstruction methods, and will not be described here.
  • the spatial position information of each picture has been restored, but these pictures are statically discrete, which is not conducive to scene display.
  • the scene to be reconstructed is better displayed. 3D features.
  • the virtual image is inserted between adjacent pictures to be rendered, and the scene to be reconstructed is generated and generated.
  • the real position information of the camera when two pictures are taken can be recovered, and in order to realize the virtual 3D excessive effect, it is necessary to insert a virtual camera position during rendering, and at the same time
  • the difference in UGC photos also requires interpolation of camera built-in parameters. Since there are common 3D points between the two frames, these points can be projected onto the virtual camera image plane, so that the mapping between the two pictures and the virtual picture can be established.
  • This embodiment filters out some ineffective and unstable feature regions, only according to the scene to be reconstructed.
  • the associated scene feature area performs three-dimensional reconstruction on the scene to be reconstructed, thereby improving reconstruction efficiency and accuracy.
  • the feature area further includes at least one of a character feature area, an item feature area, and a scene feature area.
  • the feature area included in the picture may be classified, and the feature area included in the picture is divided into a feature area, an item feature area, and a scene by using a clustering algorithm. At least one of a feature area and a scene feature area.
  • the character feature area refers to the feature area with the theme of the person
  • the item feature area refers to the feature area with the theme of the item
  • the scene feature area refers to the feature with the theme of natural scenery (for example, sky, clouds, trees, etc.)
  • the area, the scene feature area refers to a feature area located in the scene to be reconstructed and related to the scene to be reconstructed, including an attraction, a building, and the like.
  • the method further includes:
  • the character feature area, the item feature area, and the scene feature area are deleted.
  • the extracted feature area needs to be filtered.
  • a large number of ineffective and unstable feature points are extracted in the pixel regions corresponding to the trees, clouds, and characters, and the features that are not related to the scene to be reconstructed may be removed. Reduce the time spent on 3D reconstruction.
  • the invalid image in the acquired image set may be further deleted.
  • acquiring the first image set that matches the to-be-reconstructed scene includes:
  • the picture that does not meet the preset requirement includes a picture whose size does not meet the requirements, a picture whose main character is a picture, a picture with a scene (such as a tree, a cloud, a sky, etc.) as a main body, and a picture mainly represented by an item (a scenic spot souvenir). , the picture with the small shop as the main body, and the error mark picture not related to the selected scene.
  • the clustering algorithm is used to classify the pictures in the second picture set.
  • a picture may contain multiple types of information, so a picture may be Classified into multiple categories.
  • this embodiment firstly adopts the method in the following document 1 to perform preliminary segmentation of the pictures, and deletes pictures that do not meet the preset requirements, including the sky, trees, and characters.
  • Document 1 Cheng M M, Zheng S, Lin W Y, et al.
  • ImageSpirit Verbal guided image parsing [J].
  • the feature area is extracted on the picture of the remaining picture set by using the method in the following document 2.
  • the picture of the remaining picture set is clustered again according to the feature area, and then the set of pictures with the most cluster is found out, if The number of pictures in the group of picture sets exceeds the set threshold range. It can be considered that the set of pictures gathers the contents of most of the pictures of the scene to be reconstructed, and can be directly used. To do the reconstruction of the picture.
  • Document 2 Kim E, Li H, Huang XA hierarchical image clustering cosegmentation framework [C]//Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012: 686-693. (Edward Kim, Hongsheng Li, Xiaolei Huang;Collaborative segmentation framework based on multi-scale image clustering; Conference on Computer Vision and Pattern Recognition; 2012)
  • the method further includes:
  • the path planning of the to-be-reconstructed scene is performed according to the determined neighboring key frames, including:
  • the picture associated with the key frame space is selected as the transition picture.
  • the picture capturing time can be obtained from the picture header information by parsing the picture.
  • the original shooting information for some pictures has been lost during the process of transmission, and it is not possible to directly obtain pictures. Taking the time, this embodiment can divide the picture into early, middle and late by the machine learning method according to the brightness change of the picture.
  • this embodiment selects by using the shortest path planning method by calculating the rendering cost between pictures.
  • the rendering cost includes not only the spatial position of the picture, the viewpoint direction, the light field, the deformation rate, the resolution, etc., but also the temporal difference between the pictures.
  • the key frame selection rule is: the number of scene feature areas included in the selected picture exceeds a preset number of pictures, and the key frame shooting position is nearby.
  • a large number of images, in addition, keyframes are also distributed over different time frames.
  • the key frame selection method is as follows: First, a frame image is selected, and the three-dimensional point included in the image is the most, and then the next frame image is searched as the starting frame, and the condition of the next frame image selected as the key frame is The newly added 3D points are enough, and it is necessary to calculate whether there are enough other pictures in a certain spatial range of the frame, and then calculate the time difference between the frame picture and the previous frame picture. If the difference is not large, try to Look for the most different image from the picture to replace the current picture as the key frame. It should be pointed out that since the collected picture content is not necessarily rich enough, the constraints are not the same importance, and the key frames need to be included. Sufficient feature points are the most important. Through the above method of selecting a key frame, and so on, the number of key frames is gradually increased, and when the number of selected key frames satisfies a certain threshold, the selection of the key frame is ended.
  • a set of pictures needs to be searched from the key frame A to the key frame B, and the selected transition pictures are recorded as p1, p2, ... pn, and it is first determined whether there is a difference in time between the key frames A and B, if A It belongs to the picture in the morning, B belongs to the night, then the selected transition picture p needs to be in this period as much as possible, so that it can maintain good continuity in the visual, and does not switch frequently during the day and night; then if there is near the A key frame If there are enough pictures, it means that better spatial information can be displayed near the A picture, so you can select more pictures near the A picture and use better image rendering methods to highlight the three-dimensional structure of the scene.
  • the generating and generating the to-be-reconstructed scene includes:
  • the reconstructed scenes generated at different times can be rendered according to the shooting time of the pictures.
  • the foregoing embodiments also filter out some ineffective and unstable feature regions, and perform three-dimensional reconstruction on the to-be-reconstructed scene only according to the scene feature region associated with the scene to be reconstructed, thereby improving reconstruction efficiency and accuracy.
  • FIG. 2 is a schematic structural diagram of a scenario reconstruction apparatus according to Embodiment 2 of the present invention. As shown in FIG. 2, the method includes: a picture acquisition module 21, a feature extraction module 22, a feature recognition module 23, and a reconstruction module 24:
  • the image obtaining module 21 is configured to acquire a first image set that matches the scene to be reconstructed
  • the feature extraction module 22 is configured to extract at least one feature region of the image in the first image set by using a feature extraction algorithm
  • the feature recognition module 23 is configured to identify the feature area to obtain a scene feature area in the picture
  • the reconstruction module 24 is configured to perform three-dimensional reconstruction on the to-be-reconstructed scene according to the scene feature area in the picture, and render and generate the to-be-reconstructed scene.
  • the scene reconstruction apparatus in this embodiment is used to perform the scene reconstruction method described in the foregoing embodiments.
  • the technical principle and the generated technical effects are similar, and are not described here.
  • the feature area further includes at least one of a character feature area, an item feature area, and a scene feature area
  • the corresponding device further includes: a feature deletion module 25;
  • the feature deletion module 25 is configured to: after the reconstruction module 24 performs three-dimensional reconstruction on the to-be-reconstructed scene according to the scene feature area in the picture, select a character feature area, an item feature area, and a scene feature area from the picture. delete.
  • the picture obtaining module 21 is specifically configured to:
  • the device further includes: a key frame selection module 26, an adjacent key frame determination module 27, and a path planning module 28;
  • the key frame selection module 26 is configured to select a scene included in the image after the reconstruction module 24 performs three-dimensional reconstruction on the to-be-reconstructed scene according to the scene feature region in the image, and before rendering and generating the to-be-reconstructed scene.
  • the number of feature areas exceeds a preset number of pictures as key frames;
  • the neighboring key frame determining module 27 is configured to determine a proximity relationship of the key frame according to the spatial relationship of the displayed scene in the key frame picture and the picture capturing time;
  • the path planning module 28 is configured to perform path planning on the to-be-reconstructed scene according to the determined neighboring key frames.
  • the path planning module 28 is specifically configured to:
  • a transition picture in a different time period is inserted between adjacent key frames; if there is a spatial difference between adjacent key frames, the key frame is selected
  • the spatially associated picture acts as a transition picture.
  • the reconstruction module 24 is specifically configured to:
  • the scene reconstruction apparatus described in the above embodiments is also used to perform the scene reconstruction method described in the foregoing embodiments.
  • the technical principle and the generated technical effects are similar, and are not described here.
  • FIG. 3 is a schematic diagram showing the hardware structure of a terminal device for implementing scene reconstruction according to Embodiment 3 of the present invention
  • the terminal device includes one or more processors 31, a memory 32, one or more modules, and the one or more modules (for example, the picture acquisition module 21 in the scene reconstruction device shown in FIG. 2, features
  • the extraction module 22, the feature recognition module 23, the reconstruction module 24, the feature deletion module 25, the key frame selection module 26, the adjacent key frame determination module 27, and the path planning module 28) are stored in the memory 32;
  • the processor 31 and the memory 32 in the terminal device can be connected by a bus or other means, and the connection through the bus is taken as an example in FIG.
  • the three-dimensional reconstruction of the to-be-reconstructed scene is performed according to the scene feature area in the picture, and the scene to be reconstructed is generated and rendered.
  • the foregoing terminal device can perform the method provided by Embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
  • the feature area is further preferably: at least one of a character feature area, an item feature area, and a scene feature area, and the processor 31 pairs the scene to be reconstructed according to the scene feature area in the picture.
  • the character feature area, the item feature area, and the scene feature area are deleted from the picture.
  • the processor 31 searches for a second picture set related to the to-be-reconstructed scene from the Internet or a user-generated content UGC picture library by using an image recognition technology; and deletes a picture in the second picture set that does not meet the preset requirement. And using the remaining picture as the first picture set that matches the to-be-reconstructed scene.
  • the processor 31 pairs the to-be-reconstructed field according to a scene feature area in the picture. After the three-dimensional reconstruction of the scene, before the rendering of the scene to be reconstructed, the number of scene feature regions included in the selected image exceeds a preset number of pictures as a key frame; according to the spatial relationship of the displayed scene in the key frame picture and the picture capturing time, And determining a neighboring relationship of the key frame; and performing path planning on the to-be-reconstructed scene according to the determined neighboring key frame.
  • the processor 31 inserts a transition picture in a different time period between adjacent key frames; if there is space between adjacent key frames For the difference, the picture associated with the key frame space is selected as the transition picture.
  • the processor 31 renders a reconstructed scene generated at different times according to the shooting time of each picture in the first picture set.
  • An embodiment of the present invention further provides a non-volatile computer storage medium, where the computer storage medium stores one or more modules, when the one or more modules are executed by a device that performs a scene reconstruction method, The device performs the following operations:
  • the three-dimensional reconstruction of the to-be-reconstructed scene is performed according to the scene feature area in the picture, and the scene to be reconstructed is generated and rendered.
  • the feature area further preferably includes at least one of a character feature area, an item feature area, and a scene feature area, according to the scene feature area in the picture.
  • a character feature area preferably includes at least one of a character feature area, an item feature area, and a scene feature area, according to the scene feature area in the picture.
  • the character feature area, the item feature area, and the scene feature area are deleted.
  • acquiring the first image set that matches the scenario to be reconstructed is preferably:
  • the method further includes:
  • the path planning of the to-be-reconstructed scene according to the determined neighboring key frames is preferably:
  • the picture associated with the key frame space is selected as the transition picture.
  • the rendering to generate the scene to be reconstructed is preferably:

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • General Engineering & Computer Science (AREA)
  • Library & Information Science (AREA)
  • Databases & Information Systems (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Hardware Design (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)
  • Image Generation (AREA)

Abstract

一种场景重建方法、装置、终端设备及存储介质，其中方法包括：获取与待重建场景匹配的第一图片集合（11）；采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域（12）；对所述特征区域进行识别得到图片中的景物特征区域（13）；根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景（14）。滤掉了一些无效不稳定的特征区域，仅根据与所述待重建场景相关联的景物特征区域对所述待重建场景进行三维重建，从而提高了重建效率和准确性。

Description

场景重建方法、装置、终端设备及存储介质
本专利申请要求于2015年08月03日提交的、申请号为201510483318.4、申请人为百度在线网络技术(北京)有限公司、发明名称为“场景重建方法及装置”的中国专利申请的优先权，该申请的全文以引用的方式并入本申请中。
技术领域
本发明实施例涉及图像处理技术领域，尤其涉及一种场景重建方法、装置、终端设备及存储介质。
背景技术
一些历史文化古迹，随时间流逝慢慢的被消磨而风采不再，甚至会在自然灾难面前毁于一旦。随着科技发展和旅游业的盛行，普通民众通过消费级别的相机搜集了特定时刻、特定视角下这些历史文化古迹的风采，并且有很多人会选择将拍摄内容传播到互联网上，使每个人都可以领略他/她所看到的美景。
在这一背景下，反过来可通过强大的搜索引擎或特定的搜集方式，可以采集到众多用户拍摄的海量图片，这些图片会分布在一天的不同时刻，也会分布在一年的不同季节，甚至会分布在不同的年份里。基于此，可通过图片筛选和重建，让公众从一个大的时间尺度和空间尺度上领略这些古迹的不同风采。
但是，由于搜集到的图片中存在大量杂图和无效图片，现有技术需要进行人工剔除，这耗费巨大的人力成本。并且，重建过程随图片数据量的增长重建时间呈指数级增长，重建效率较低。
发明内容
本发明实施例提供一种场景重建方法、装置、终端设备及存储介质，能够提高重建效率。
第一方面，本发明实施例提供了一种场景重建方法，包括：
获取与待重建场景匹配的第一图片集合；
采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
对所述特征区域进行识别得到图片中的景物特征区域；
根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
第二方面，本发明实施例还提供一种场景重建装置，包括：
图片获取模块，用于获取与待重建场景匹配的第一图片集合；
特征提取模块，用于采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
特征识别模块，用于对所述特征区域进行识别得到图片中的景物特征区域；
重建模块，用于根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
第三方面，本发明实施例还提供一种实现场景重建的终端设备，包括：
一个或者多个处理器；
存储器；
一个或者多个模块，所述一个或者多个模块存储在所述存储器中，当被所述一个或者多个处理器执行时，进行如下操作：
获取与待重建场景匹配的第一图片集合；
采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
对所述特征区域进行识别得到图片中的景物特征区域；
根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
第四方面，本发明实施例还提供一种非易失性计算机存储介质，所述计算机存储介质存储有一个或者多个模块，当所述一个或者多个模块被一个执行场景重建方法的设备执行时，使得所述设备执行如下操作：
获取与待重建场景匹配的第一图片集合；
采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
对所述特征区域进行识别得到图片中的景物特征区域；
根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。本发明实施例的技术方案，滤掉了一些无效不稳定的特征区域，仅根据与所述待重建场景相关联的景物特征区域对所述待重建场景进行三维重建，从而提高了重建效率和准确性。
附图说明
图1为本发明实施例一提供的场景重建方法的流程示意图；
图2为本发明实施例二提供的场景重建装置的结构示意图；
图3为本发明实施例三提供的实现场景重建的终端设备的结构示意图。
具体实施方式
下面结合附图和实施例对本发明作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本发明，而非对本发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本发明相关的部分而非全部结构。
本发明实施例提供的场景重建方法的执行主体，可为本发明实施例提供的场景重建装置，或者集成了所述场景重建装置的服务器设备，该场景重建装置可以采用硬件或软件实现。
实施例一
图1为本发明实施例一提供的场景重建方法的流程示意图，如图1所示，具体包括：
S11、获取与待重建场景匹配的第一图片集合；
其中，在本实施例中，所述待重建场景可为一些旅游景点、历史古迹和建筑物等。所述第一图片集合中包含至少一张与所述待重建场景匹配的图片。具体可通过在互联网上输入与所述待重建场景相关的关键字搜索得到，也可从用户生成内容(User Generated Content，UGC)图片库中获取。
其中，UGC图片库中保存了用户在互联网上分享的图片内容，这些图片内容大都来源于用户拍摄的照片，也可以是用户制作的图片。由于市面上相机的多样性，而且由于拍摄时间、地点以及方式的不同，UGC图片库中记录了同一场景在不同的视点、时间(四季变换或早晚交替)、光照条件(阴、晴、雨或雪)下的不同外观，随着用户上传数据量的急速增长，UGC图片库具备覆盖面广，数据获取成本低等特点。因此，从所述UGC图片库中可获取到比较有价值的图 片内容。
S12、采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
其中，可采用特征提取算法包括角点检测算法和局部不变特征点提取方法等。
另外，为增强图片中特征的独立性和有效性，在对图片特征提取之前，需要对图片进行的前期处理，主要的处理为图片特征分割。即将图片分成若干个特定的、具有独特性质的区域，并提出目标对象(例如，图片中的人物区域、景色区域和景物区域等)。在本实施例中，可采用的图像分割方法主要有基于阈值的分割方法、基于区域的分割方法、基于边缘的分割方法以及基于特定理论的分割方法等。
S13、对所述特征区域进行识别得到图片中的景物特征区域；
在上述步骤S12中，提取出图片中的特征区域之后，需要对特征区域进行识别，以便后续区分特征区域，确定所述图片中与所述待重建场景匹配的特征区域，即景物特征区域。在此步骤中可通过以下特征识别算法对图片特征区域进行识别：基于图统计特征的图像识别算法、基于HOG特征的目标识别算法研究和基于局部特征的物体识别算法等。
S14、根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
其中，景物特征区域可以是与所述待重建场景中某景点匹配的场景的轮廓。以所述待重建场景中景点为龙门石窟为例，可以从龙门石窟中的佛像的多个对象(例如佛像的脸部、眼部、手部)的轮廓上选取特征区域。
具体三维重建过程可采用现有的一些重建方法，这里不再赘述。通过三维重建，已恢复出每张图片所处空间位置信息，但这些图片是静态离散的不利于场景展示，为保证这些图片空间上具有较好的延续性，更好的展现所述待重建场景的三维特征。本实施例图片渲染技术，通过在待渲染的相邻图片间插入虚拟图片，渲染生成所述待重建场景。
具体的，通过之前的图片特征区域及三维重建技术，可恢复出其中两张图片拍照时的相机的真实位置信息，为实现虚拟的3D过度效果，需要在渲染时插入虚拟的相机位置，同时由于UGC相片的差异性，也需要对相机内置参数进行内插处理。由于两帧图片间存在共有的三维点，这些点可以投影到虚拟的相机成像面上，如此就可以建立两张图片与虚拟图片之间的映射关系。例如，在当某一图片A附近有较多在空间上相关联的图片B时，但图片中的场景的三维结构没有明显前后之分的时候，对于待渲染的两个图片A和B，将它们之间存在的公共三维点投射到虚拟的相机成像面上，而后分别在图片A、B及虚拟的相机成像面上的特征区域进行三角构网，这样图片A、B及虚拟的相机成像面上会形成一一对应的三角形区域，可以认为三角形内部是平面，而后在虚拟的相机成像面上，根据三角形区域的对应关系，分别从图片A、B上取像素点填充到虚拟的相机成像面上的三角形区域中，这样就生成完整的虚拟图片，如此过渡，可以生成三维空间结构较准确且细节丰富的过渡效果。
在渲染过程中，由于虚拟相机在两个真实相机之间移动，当其靠近某一真实相机时，该相机投射到虚拟相机上产生的形变较小，可通过设置不同的权重，使虚拟过度的效果达到最佳。
本实施例滤掉了一些无效不稳定的特征区域，仅根据与所述待重建场景相 关联的景物特征区域对所述待重建场景进行三维重建，从而提高了重建效率和准确性。
示例性的，所述特征区域还包括：人物特征区域、物品特征区域和景色特征区域中的至少一种。
具体的，在对所述特征区域进行识别的过程中，可对所述图片包含的特征区域进行分类，采用聚类算法将所述图片包含的特征区域划分为人物特征区域、物品特征区域、景色特征区域和景物特征区域中的至少一种。
其中，人物特征区域是指以人物为主题的特征区域，物品特征区域是指以物品为主题的特征区域，景色特征区域是指以自然景色(例如，天空、云朵和树木等)为主题的特征区域，景物特征区域是指位于所述待重建场景中且与所述待重建场景相关的特征区域，包括景点、建筑物等。
相应的，为减小三维重建的数据量，提高重建效率，在根据所述图片中的景物特征区域对所述待重建场景进行三维重建之前，还包括：
从图片中，将人物特征区域、物品特征区域和景色特征区域删除。
具体的，在提取出图片中的特征区域之后，为减小三维重建的数据量，提高重建效率，需要对提取特征区域进行过滤。具体来说，当图片上存在树木、人物和云朵时，会在树木、云朵、人物对应的像素区域提取大量无效不稳定的特征点，可将这些与所述待重建场景无关的特征去除，以减少三维重建消耗的时间。
示例性的，为进一步减少处理处理数据量，可进一步删除获取的图片集合中的无效图片，具体的，获取与待重建场景匹配的第一图片集合包括：
采用图像识别技术从互联网或用户生成内容UGC图片库中搜索与所述待重建场景相关的第二图片集合；
删除第二图片集合中不符合预设要求的图片，将剩余图片作为与所述待重建场景匹配的第一图片集合。
其中，所述不符合预设要求的图片包括尺寸不符合要求的图片、人物为主体的图片、景色(例如树木、云朵和天空等)为主体的图片、以物品(景区纪念品)为主体的图片、以小卖铺为主体的图片、及与所选景物无关的错误标记图片等。
具体的，采用聚类算法对上述第二图片集合中的图片进行分类，在分类过程中，由于图片中包含场景的复杂性，一张图片中可能包含多类信息，因此一张图片可能会被归为多个类别中。为避免删除一些有用的图片，确保分类的准确性，本实施例首先采用如下文献一中的方法对图片做初步分割，删除不符合预设要求的图片，包括以天空、树木、人物占主体的图片。文献一：Cheng M M，Zheng S，Lin W Y，et al.ImageSpirit：Verbal guided image parsing[J].ACM Transactions on Graphics(TOG)，2014，34(1)：3.(Ming-Ming Cheng，Shuai Zheng，Wen-Yan Lin，Vibhav Vineet，Paul Sturgess，Nigel Crook，Niloy Mitra，Philip Torr；ImageSpirit：基于语义指导的图像解析，美国计算机学会图形学报(TOG)；2014)
然后，采用如下文献二中的方法在剩余图片集合的图片上提取特征区域，根据特征区域对剩余图片集合的图片再次做聚类处理，而后从中找出聚类最多的一组图片集合，若该组图片集合中的图片数量超出设定的阈值范围，可以认为这一组图片集合聚集了所述待重建场景的绝大多数图片的内容，可以直接用 来做重建图片。文献二：Kim E，Li H，Huang X.A hierarchical image clustering cosegmentation framework[C]//Computer Vision and Pattern Recognition(CVPR)，2012IEEE Conference on.IEEE，2012：686-693.(Edward Kim，Hongsheng Li，Xiaolei Huang；基于多尺度图像聚类的协同分割框架；计算机视觉与模式识别学术会议；2012)
但也有可能对于同一景物对应多个拍摄视角，也就意味着有可能会出现多个聚类中心，此时需使用特征匹配算法判断类别间的连接性，若某一组图片集合中图片的特征与上述聚类最多的一组图片集合特征匹配的数目多出设定阈值，将该组图片集合也用来做重建图片。
示例性的，根据所述图片中的景物特征区域对所述待重建场景进行三维重建之后，渲染生成所述待重建场景之前，还包括：
选取图片中包括的景物特征区域的数量超过预设数量的图片作为关键帧；
根据关键帧图片中显示场景的空间关系和图片拍摄时间，确定关键帧的邻近关系；
根据确定的邻近关键帧对所述待重建场景进行路径规划。
示例性的，根据确定的邻近关键帧对所述待重建场景进行路径规划，包括：
若相邻关键帧之间存在时间上的差异，则在相邻关键帧之间插入差异时间段内的过渡图片；
若相邻关键帧之间存在空间上的差异，则选取与所述关键帧空间上相关联图片作为过渡图片。
其中，图片拍摄时间可通过解析所述图片，从图片头信息中获取。另外，对于一些图片的原始拍摄信息在传播过程中已经损失掉，无法直接获取图片拍 摄时间，本实施例可根据图片亮度变化，通过机器学习方法，将图片划分成早、中、晚三类。
由于上述参与重建的图片集合中的图片，在空间上分布并不均匀，而且这些图片在时间上也没有明确分布规律。
为从大量图片中选取一组在空间上展现最为丰富的图片，本实施例通过计算图片间的渲染代价，使用最短路径规划方法来选取。为此渲染代价不仅包含图片的空间位置、视点方向、光场、形变率、分辨率等内容，同时还计算图片间在时间上的差异性。
具体的，首先需要从第一图片集合中选出一些图片作为关键帧，关键帧的选取规则是：选取图片中包括的景物特征区域的数量超过预设数量的图片，同时关键帧拍摄位置附近有大量的图片，此外，关键帧也要分布在不同时间范围内。
关键帧选取方法具体如下：首先选取一帧图片，该图片上包含的三维点是最多的，而后以此为起始帧搜索下一帧图像，下一帧图像的被选为关键帧的条件是，新增加的三维点足够多，同时需要计算该帧一定空间范围内是否有足够多的其他图片，之后计算该帧图片与上一帧图片在时间上的差异性，如果差异不大，可以尽量从该图片附近找差异性较大的图片替代当前图片作为关键帧，需要指出的是，由于搜集到的图片内容并不一定足够丰富，因此各个约束条件重要性并不相同，其中关键帧需要包含足够多的特征点最为重要。通过上述选取关键帧方法，以此类推，逐步增加关键帧数量，当选取的关键帧数量满足一定阈值后就结束关键帧的选取。
对于选取得到的关键帧，只能大致覆盖重建场景，但相邻关键帧之前并不 能直接通过影像渲染技术进行过渡，因此在关键帧之间需要找一组影像以实现关键帧之间的平滑过渡，另外由于关键帧之间有一定的差异性，在选取图片时需尽量使用图片间的差异性。
具体的，假定需要从关键帧A找出一组图片渲染到关键帧B，选取的过渡图片记为p1、p2…pn，首先判断关键帧A与B之间是否存在时间上的差异，假如A属于早上的图片，B属于晚上，则选取的过渡图片p需尽量处于这一段时间内，这样在视觉上可以保持较好的延续性，而不至于白天夜晚切换频繁；而后如果A关键帧附近有足够多图片，则以为着A图片附近可以展示较好的空间信息，因此可以在A图片附近选取较多图片，并使用较好的图像渲染方法，突出场景的三维结构。
示例性的，渲染生成所述待重建场景包括：
根据所述第一图片集合中各图片的拍摄时间，渲染生成不同时间的重建场景。
由于第一图像集合中的图片拍摄于不同时间，甚至于不同的时期，因此可以根据图片的拍摄时间，渲染生成不同时间的重建场景。
上述各实施例同样滤掉了一些无效不稳定的特征区域，仅根据与所述待重建场景相关联的景物特征区域对所述待重建场景进行三维重建，从而提高了重建效率和准确性。
并且通过使用路径规划技术，进一步保证了渲染时图片的连续性，提高视觉效果。
实施例二
图2为本发明实施例二提供的场景重建装置的结构示意图，如图2所示，具体包括：图片获取模块21、特征提取模块22、特征识别模块23和重建模块24：
所述图片获取模块21用于获取与待重建场景匹配的第一图片集合；
所述特征提取模块22用于采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
所述特征识别模块23用于对所述特征区域进行识别得到图片中的景物特征区域；
所述重建模块24用于根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
本实施例所述的场景重建装置用于执行上述各实施例所述的场景重建方法，其技术原理和产生的技术效果类似，这里不再累述。
示例性的，在上述实施例的基础上，所述特征区域还包括人物特征区域、物品特征区域和景色特征区域中的至少一种，相应的所述装置还包括：特征删除模块25；
所述特征删除模块25用于在所述重建模块24根据所述图片中的景物特征区域对所述待重建场景进行三维重建之前，从图片中，将人物特征区域、物品特征区域和景色特征区域删除。
示例性的，在上述实施例的基础上，所述图片获取模块21具体用于：
采用图像识别技术从互联网或用户生成内容UGC图片库中搜索与所述待重建场景相关的第二图片集合；删除第二图片集合中不符合预设要求的图片， 将剩余图片作为与所述待重建场景匹配的第一图片集合。
示例性的，在上述实施例的基础上，所述装置还包括：关键帧选取模块26、邻近关键帧确定模块27和路径规划模块28；
所述关键帧选取模块26用于在所述重建模块24根据所述图片中的景物特征区域对所述待重建场景进行三维重建之后，渲染生成所述待重建场景之前，选取图片中包括的景物特征区域的数量超过预设数量的图片作为关键帧；
所述邻近关键帧确定模块27用于根据关键帧图片中显示场景的空间关系和图片拍摄时间，确定关键帧的邻近关系；
所述路径规划模块28用于根据确定的邻近关键帧对所述待重建场景进行路径规划。
示例性的，在上述实施例的基础上，所述路径规划模块28具体用于：
若相邻关键帧之间存在时间上的差异，则在相邻关键帧之间插入差异时间段内的过渡图片；若相邻关键帧之间存在空间上的差异，则选取与所述关键帧空间上相关联图片作为过渡图片。
示例性的，在上述实施例的基础上，所述重建模块24具体用于：
根据所述第一图片集合中各图片的拍摄时间，渲染生成不同时间的重建场景。
上述各实施例所述的场景重建装置同样用于执行上述各实施例所述的场景重建方法，其技术原理和产生的技术效果类似，这里不再累述。
实施例三
图3为本发明实施例三提供的一种实现场景重建的终端设备的硬件结构示 意图，该终端设备包括一个或多个处理器31、存储器32，一个或者多个模块，所述一个或者多个模块(例如，附图2所示的场景重建装置中的图片获取模块21、特征提取模块22、特征识别模块23、重建模块24、特征删除模块25、关键帧选取模块26、邻近关键帧确定模块27和路径规划模块28)存储在所述存储器32中；图3中以一个处理器31为例；终端设备中的处理器31和存储器32可以通过总线或其他方式连接，图3中以通过总线连接为例。
当被所述一个或者多个处理器31执行时，进行如下操作：
获取与待重建场景匹配的第一图片集合；
采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
对所述特征区域进行识别得到图片中的景物特征区域；
根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
上述终端设备可执行本发明实施例一所提供的方法，具备执行方法相应的功能模块和有益效果。
示例性的，所述特征区域还优选为：人物特征区域、物品特征区域和景色特征区域中的至少一种，则所述处理器31根据所述图片中的景物特征区域对所述待重建场景进行三维重建之前，从图片中，将人物特征区域、物品特征区域和景色特征区域删除。
示例性的，所述处理器31采用图像识别技术从互联网或用户生成内容UGC图片库中搜索与所述待重建场景相关的第二图片集合；删除第二图片集合中不符合预设要求的图片，将剩余图片作为与所述待重建场景匹配的第一图片集合。
示例性的，所述处理器31根据所述图片中的景物特征区域对所述待重建场 景进行三维重建之后，渲染生成所述待重建场景之前，选取图片中包括的景物特征区域的数量超过预设数量的图片作为关键帧；根据关键帧图片中显示场景的空间关系和图片拍摄时间，确定关键帧的邻近关系；根据确定的邻近关键帧对所述待重建场景进行路径规划。
示例性的，所述处理器31若相邻关键帧之间存在时间上的差异，则在相邻关键帧之间插入差异时间段内的过渡图片；若相邻关键帧之间存在空间上的差异，则选取与所述关键帧空间上相关联图片作为过渡图片。
示例性的，所述处理器31根据所述第一图片集合中各图片的拍摄时间，渲染生成不同时间的重建场景。
实施例四
本发明实施例还提供一种非易失性计算机存储介质，所述计算机存储介质存储有一个或者多个模块，当所述一个或者多个模块被一个执行场景重建方法的设备执行时，使得所述设备执行如下操作：
获取与待重建场景匹配的第一图片集合；
采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
对所述特征区域进行识别得到图片中的景物特征区域；
根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
上述存储介质中存储的模块被所述设备所执行时，所述特征区域还优选包括：人物特征区域、物品特征区域和景色特征区域中的至少一种，则根据所述图片中的景物特征区域对所述待重建场景进行三维重建之前，还优选包括：
从图片中，将人物特征区域、物品特征区域和景色特征区域删除。
上述存储介质中存储的模块被所述设备所执行时，获取与待重建场景匹配的第一图片集合优选为：
采用图像识别技术从互联网或用户生成内容UGC图片库中搜索与所述待重建场景相关的第二图片集合；
删除第二图片集合中不符合预设要求的图片，将剩余图片作为与所述待重建场景匹配的第一图片集合。
上述存储介质中存储的模块被所述设备所执行时，根据所述图片中的景物特征区域对所述待重建场景进行三维重建之后，渲染生成所述待重建场景之前，还优选包括：
选取图片中包括的景物特征区域的数量超过预设数量的图片作为关键帧；
根据关键帧图片中显示场景的空间关系和图片拍摄时间，确定关键帧的邻近关系；
根据确定的邻近关键帧对所述待重建场景进行路径规划。
上述存储介质中存储的模块被所述设备所执行时，根据确定的邻近关键帧对所述待重建场景进行路径规划优选为：
若相邻关键帧之间存在时间上的差异，则在相邻关键帧之间插入差异时间段内的过渡图片；
若相邻关键帧之间存在空间上的差异，则选取与所述关键帧空间上相关联图片作为过渡图片。
上述存储介质中存储的模块被所述设备所执行时，渲染生成所述待重建场景优选为：
根据所述第一图片集合中各图片的拍摄时间，渲染生成不同时间的重建场 景。
注意，上述仅为本发明的较佳实施例及所运用技术原理。本领域技术人员会理解，本发明不限于这里所述的特定实施例，对本领域技术人员来说能够进行各种明显的变化、重新调整和替代而不会脱离本发明的保护范围。因此，虽然通过以上实施例对本发明进行了较为详细的说明，但是本发明不仅仅限于以上实施例，在不脱离本发明构思的情况下，还可以包括更多其他等效实施例，而本发明的范围由所附的权利要求范围决定。

Claims (14)

  1. 一种场景重建方法，其特征在于，包括：
    获取与待重建场景匹配的第一图片集合；
    采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
    对所述特征区域进行识别得到图片中的景物特征区域；
    根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
  2. 根据权利要求1所述的方法，其特征在于，所述特征区域还包括：人物特征区域、物品特征区域和景色特征区域中的至少一种，则根据所述图片中的景物特征区域对所述待重建场景进行三维重建之前，还包括：
    从图片中，将人物特征区域、物品特征区域和景色特征区域删除。
  3. 根据权利要求1所述的方法，其特征在于，获取与待重建场景匹配的第一图片集合，包括：
    采用图像识别技术从互联网或用户生成内容UGC图片库中搜索与所述待重建场景相关的第二图片集合；
    删除第二图片集合中不符合预设要求的图片，将剩余图片作为与所述待重建场景匹配的第一图片集合。
  4. 根据权利要求1～3任一项所述的方法，其特征在于，根据所述图片中的景物特征区域对所述待重建场景进行三维重建之后，渲染生成所述待重建场景之前，还包括：
    选取图片中包括的景物特征区域的数量超过预设数量的图片作为关键帧；
    根据关键帧图片中显示场景的空间关系和图片拍摄时间，确定关键帧的邻近关系；
    根据确定的邻近关键帧对所述待重建场景进行路径规划。
  5. 根据权利要求4所述的方法，其特征在于，根据确定的邻近关键帧对所述待重建场景进行路径规划，包括：
    若相邻关键帧之间存在时间上的差异，则在相邻关键帧之间插入差异时间段内的过渡图片；
    若相邻关键帧之间存在空间上的差异，则选取与所述关键帧空间上相关联图片作为过渡图片。
  6. 根据权利要求1所述的方法，其特征在于，渲染生成所述待重建场景包括：
    根据所述第一图片集合中各图片的拍摄时间，渲染生成不同时间的重建场景。
  7. 一种场景重建装置，其特征在于，包括：
    图片获取模块，用于获取与待重建场景匹配的第一图片集合；
    特征提取模块，用于采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
    特征识别模块，用于对所述特征区域进行识别得到图片中的景物特征区域；
    重建模块，用于根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
  8. 根据权利要求7所述的装置，其特征在于，所述特征区域还包括：人物特征区域、物品特征区域和景色特征区域中的至少一种；
    相应的，所述装置还包括：
    特征删除模块，用于在所述重建模块根据所述图片中的景物特征区域对所 述待重建场景进行三维重建之前，从图片中，将人物特征区域、物品特征区域和景色特征区域删除。
  9. 根据权利要求7所述的装置，其特征在于，所述图片获取模块具体用于：
    采用图像识别技术从互联网或用户生成内容UGC图片库中搜索与所述待重建场景相关的第二图片集合；删除第二图片集合中不符合预设要求的图片，将剩余图片作为与所述待重建场景匹配的第一图片集合。
  10. 根据权利要求7～9任一项所述的装置，其特征在于，还包括：
    关键帧选取模块，用于在所述重建模块根据所述图片中的景物特征区域对所述待重建场景进行三维重建之后，渲染生成所述待重建场景之前，选取图片中包括的景物特征区域的数量超过预设数量的图片作为关键帧；
    邻近关键帧确定模块，用于根据关键帧图片中显示场景的空间关系和图片拍摄时间，确定关键帧的邻近关系；
    路径规划模块，用于根据确定的邻近关键帧对所述待重建场景进行路径规划。
  11. 根据权利要求10所述的装置，其特征在于，所述路径规划模块具体用于：
    若相邻关键帧之间存在时间上的差异，则在相邻关键帧之间插入差异时间段内的过渡图片；若相邻关键帧之间存在空间上的差异，则选取与所述关键帧空间上相关联图片作为过渡图片。
  12. 根据权利要求7所述的装置，其特征在于，所述重建模块具体用于：
    根据所述第一图片集合中各图片的拍摄时间，渲染生成不同时间的重建场景。
  13. 一种实现场景重建的终端设备，其特征在于，包括：
    一个或者多个处理器；
    存储器；
    一个或者多个模块，所述一个或者多个模块存储在所述存储器中，当被所述一个或者多个处理器执行时，进行如下操作：
    获取与待重建场景匹配的第一图片集合；
    采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
    对所述特征区域进行识别得到图片中的景物特征区域；
    根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
  14. 一种非易失性计算机存储介质，所述计算机存储介质存储有一个或者多个模块，其特征在于，当所述一个或者多个模块被一个执行场景重建方法的设备执行时，使得所述设备执行如下操作：
    获取与待重建场景匹配的第一图片集合；
    采用特征提取算法提取所述第一图片集合中图片的至少一个特征区域；
    对所述特征区域进行识别得到图片中的景物特征区域；
    根据所述图片中的景物特征区域对所述待重建场景进行三维重建，并渲染生成所述待重建场景。
PCT/CN2015/096623 2015-08-03 2015-12-08 场景重建方法、装置、终端设备及存储介质 WO2017020467A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2018506162A JP6553285B2 (ja) 2015-08-03 2015-12-08 シーン再構成方法、装置、端末機器及び記憶媒体
US15/749,642 US10467800B2 (en) 2015-08-03 2015-12-08 Method and apparatus for reconstructing scene, terminal device, and storage medium
EP15900236.9A EP3324368B1 (en) 2015-08-03 2015-12-08 Scenario reconstruction method and apparatus, terminal device, and storage medium
KR1020187005930A KR102033262B1 (ko) 2015-08-03 2015-12-08 정경 재구성 방법, 장치, 단말기 장치 및 저장 매체

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510483318.4A CN105139445B (zh) 2015-08-03 2015-08-03 场景重建方法及装置
CN201510483318.4 2015-08-03

Publications (1)

Publication Number Publication Date
WO2017020467A1 true WO2017020467A1 (zh) 2017-02-09

Family

ID=54724777

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/096623 WO2017020467A1 (zh) 2015-08-03 2015-12-08 场景重建方法、装置、终端设备及存储介质

Country Status (6)

Country Link
US (1) US10467800B2 (zh)
EP (1) EP3324368B1 (zh)
JP (1) JP6553285B2 (zh)
KR (1) KR102033262B1 (zh)
CN (1) CN105139445B (zh)
WO (1) WO2017020467A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107657660A (zh) * 2017-09-15 2018-02-02 南开大学 一种基于等距正视摄影测量约束的无人机视觉快速三维重建方法
CN112750203A (zh) * 2021-01-21 2021-05-04 脸萌有限公司 模型重建方法、装置、设备及存储介质
CN114549766A (zh) * 2022-04-24 2022-05-27 成都纵横自动化技术股份有限公司 一种实时ar可视化方法、装置、设备及存储介质

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017180615A1 (en) 2016-04-12 2017-10-19 Quidient, Llc Quotidian scene reconstruction engine
CN108030452A (zh) * 2017-11-30 2018-05-15 深圳市沃特沃德股份有限公司 视觉扫地机器人及建立场景地图的方法
US11875476B2 (en) 2018-05-02 2024-01-16 Quidient, Llc Codec for processing scenes of almost unlimited detail
CN108765487B (zh) 2018-06-04 2022-07-22 百度在线网络技术（北京）有限公司 重建三维场景的方法、装置、设备和计算机可读存储介质
CN109934777B (zh) * 2019-01-09 2023-06-02 深圳市三宝创新智能有限公司 图像局部不变特征提取方法、装置、计算机设备及存储介质
CN110134801A (zh) * 2019-04-28 2019-08-16 福建星网视易信息系统有限公司 一种作品名称与多媒体文件的匹配方法及存储介质
CN112446961A (zh) * 2019-08-30 2021-03-05 中兴通讯股份有限公司 场景重建系统及方法
CN111383337B (zh) * 2020-03-20 2023-06-27 北京百度网讯科技有限公司 用于识别对象的方法和装置
CN111556291B (zh) * 2020-04-23 2021-07-13 东电创新(北京)科技发展股份有限公司 一种便携式一体化视频采集及回传系统
CN113050643B (zh) * 2021-03-19 2024-06-21 京东鲲鹏(江苏)科技有限公司 无人车路径规划方法、装置、电子设备和计算机可读介质
US11961184B2 (en) * 2021-09-17 2024-04-16 Samsung Electronics Co., Ltd. System and method for scene reconstruction with plane and surface reconstruction
CN115222896B (zh) * 2022-09-20 2023-05-23 荣耀终端有限公司 三维重建方法、装置、电子设备及计算机可读存储介质
CN116452776B (zh) * 2023-06-19 2023-10-20 国网浙江省电力有限公司湖州供电公司 基于视觉同步定位与建图系统的低碳变电站场景重建方法
CN117538881B (zh) * 2024-01-10 2024-05-07 海底鹰深海科技股份有限公司 一种声呐水体成像波束形成方法、系统、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101777059A (zh) * 2009-12-16 2010-07-14 中国科学院自动化研究所 一种提取地标性场景摘要的方法
CN101984463A (zh) * 2010-11-02 2011-03-09 中兴通讯股份有限公司 全景图合成方法及装置
CN103139479A (zh) * 2013-02-25 2013-06-05 广东欧珀移动通信有限公司 一种结束全景预览扫描方法及装置
CN104699842A (zh) * 2015-03-31 2015-06-10 百度在线网络技术（北京）有限公司 图片展示方法和装置

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5598515A (en) * 1994-01-10 1997-01-28 Gen Tech Corp. System and method for reconstructing surface elements of solid objects in a three-dimensional scene from a plurality of two dimensional images of the scene
JP3516942B2 (ja) * 2001-11-26 2004-04-05 ザイオソフト株式会社 ３次元画像処理方法、装置およびプログラム
US7142726B2 (en) * 2003-03-19 2006-11-28 Mitsubishi Electric Research Labs, Inc. Three-dimensional scene reconstruction from labeled two-dimensional images
JP4779479B2 (ja) * 2005-07-15 2011-09-28 大日本印刷株式会社 物体の三次元形状データの削減装置および削減方法
KR100721536B1 (ko) * 2005-12-09 2007-05-23 한국전자통신연구원 2차원 평면상에서 실루엣 정보를 이용한 3차원 구조 복원방법
EP1997070B1 (de) * 2006-03-22 2018-01-03 Pilz GmbH & Co. KG Raum-zeitliches verfahren und vorrichtung zur korrespondenzbestimmung, vorzugsweise zur dreidimensionalen rekonstruktion einer szene
US8655052B2 (en) * 2007-01-26 2014-02-18 Intellectual Discovery Co., Ltd. Methodology for 3D scene reconstruction from 2D image sequences
JP4963105B2 (ja) 2007-11-22 2012-06-27 インターナショナル・ビジネス・マシーンズ・コーポレーション 画像を記憶する方法、装置
CN101739721B (zh) * 2010-01-21 2012-01-11 西北工业大学 基于时变和无序多图像的四维场景建模方法
JP5821526B2 (ja) 2011-10-27 2015-11-24 ソニー株式会社 画像処理装置、画像処理方法及びプログラム
KR101706216B1 (ko) * 2012-04-03 2017-02-13 한화테크윈 주식회사 고밀도 삼차원 영상 재구성 장치 및 방법
KR101443327B1 (ko) * 2013-01-02 2014-09-26 주식회사 오렌지큐브 실시간 렌더링 3d 오브젝트 인터렉티브 멀티비전 시스템 및 그의 실시간 렌더링 3d 오브젝트 처리방법
CN104463962B (zh) * 2014-12-09 2017-02-22 合肥工业大学 基于gps信息视频的三维场景重建方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101777059A (zh) * 2009-12-16 2010-07-14 中国科学院自动化研究所 一种提取地标性场景摘要的方法
CN101984463A (zh) * 2010-11-02 2011-03-09 中兴通讯股份有限公司 全景图合成方法及装置
CN103139479A (zh) * 2013-02-25 2013-06-05 广东欧珀移动通信有限公司 一种结束全景预览扫描方法及装置
CN104699842A (zh) * 2015-03-31 2015-06-10 百度在线网络技术（北京）有限公司 图片展示方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3324368A4 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107657660A (zh) * 2017-09-15 2018-02-02 南开大学 一种基于等距正视摄影测量约束的无人机视觉快速三维重建方法
CN112750203A (zh) * 2021-01-21 2021-05-04 脸萌有限公司 模型重建方法、装置、设备及存储介质
CN112750203B (zh) * 2021-01-21 2023-10-31 脸萌有限公司 模型重建方法、装置、设备及存储介质
CN114549766A (zh) * 2022-04-24 2022-05-27 成都纵横自动化技术股份有限公司 一种实时ar可视化方法、装置、设备及存储介质

Also Published As

Publication number Publication date
EP3324368A1 (en) 2018-05-23
KR102033262B1 (ko) 2019-10-16
US10467800B2 (en) 2019-11-05
EP3324368B1 (en) 2019-10-23
CN105139445B (zh) 2018-02-13
JP2018526736A (ja) 2018-09-13
US20180225864A1 (en) 2018-08-09
JP6553285B2 (ja) 2019-07-31
KR20180035869A (ko) 2018-04-06
CN105139445A (zh) 2015-12-09
EP3324368A4 (en) 2018-05-23

Similar Documents

Publication Publication Date Title
WO2017020467A1 (zh) 场景重建方法、装置、终端设备及存储介质
Li et al. Reconstructing building mass models from UAV images
Kelly et al. BigSUR: Large-scale structured urban reconstruction
Kyriakaki et al. 4D reconstruction of tangible cultural heritage objects from web-retrieved images
Lalonde et al. Photo clip art
Agarwal et al. Building rome in a day
Chen et al. Poseshop: Human image database construction and personalized content synthesis
Raguram et al. Modeling and recognition of landmark image collections using iconic scene graphs
US20150332117A1 (en) Composition modeling for photo retrieval through geometric image segmentation
Li et al. Diminished reality using appearance and 3D geometry of internet photo collections
CN103366400A (zh) 一种三维头像自动生成方法
US20210272253A1 (en) Automatically merging people and objects from multiple digital images to generate a composite digital image
CN107194985A (zh) 一种面向大场景的三维可视化方法及装置
Lopez et al. Modeling complex unfoliaged trees from a sparse set of images
Doulamis et al. 4D modelling in cultural heritage
Bulbul et al. Social media based 3D visual popularity
Diaz et al. Detecting dynamic objects with multi-view background subtraction
Liu et al. Stereo video object segmentation using stereoscopic foreground trajectories
CN107871338B (zh) 基于场景装饰的实时交互渲染方法
Xiao et al. Enhanced 3-D modeling for landmark image classification
Ali et al. Single image Façade segmentation and computational rephotography of House images using deep learning
Zhou et al. Modeling perspective effects in photographic composition
Delussu et al. Synthetic Data for Video Surveillance Applications of Computer Vision: A Review
Doulamis Automatic 3D reconstruction from unstructured videos combining video summarization and structure from motion
Garro et al. Label transfer exploiting three-dimensional structure for semantic segmentation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15900236

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15749642

Country of ref document: US

ENP Entry into the national phase

Ref document number: 2018506162

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20187005930

Country of ref document: KR

Kind code of ref document: A