WO2018214086A1 - 场景的三维重建方法、装置及终端设备 - Google Patents

场景的三维重建方法、装置及终端设备 Download PDF

Info

Publication number
WO2018214086A1
WO2018214086A1 PCT/CN2017/085824 CN2017085824W WO2018214086A1 WO 2018214086 A1 WO2018214086 A1 WO 2018214086A1 CN 2017085824 W CN2017085824 W CN 2017085824W WO 2018214086 A1 WO2018214086 A1 WO 2018214086A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature
feature point
frame
current frame
plane
Prior art date
Application number
PCT/CN2017/085824
Other languages
English (en)
French (fr)
Inventor
程俊
潘亮亮
姬晓鹏
王鹏
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Priority to PCT/CN2017/085824 priority Critical patent/WO2018214086A1/zh
Publication of WO2018214086A1 publication Critical patent/WO2018214086A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

Definitions

  • the invention belongs to the technical field of three-dimensional scene reconstruction, and particularly relates to a method, a device and a terminal device for three-dimensional reconstruction of a scene.
  • 3D reconstruction with RGB-D cameras has a unique advantage over binocular cameras because RGB-D cameras can provide depth values corresponding to image pixel coordinates without spending a lot of computational resources on pixel parallax calculations.
  • the traditional 3D reconstruction method based on RGB-D camera mainly uses feature points to detect and match features.
  • three-dimensional reconstruction based on RGB-D camera is mainly based on the scene observed by the camera at multiple angles, and the camera's camera pose is calculated by using the spatial positional relationship of the feature to restore the three-dimensional information of the scene.
  • the 3D reconstruction algorithm based on RGB-D camera is mainly divided into the following steps: First, the feature points are detected and extracted from the acquired image, and the corresponding depth map is used to obtain the spatial coordinates of the corresponding feature points. Secondly, the acquired feature points are detected and extracted every frame, and the detected feature points are matched with the points on the map to calculate the camera pose of the current camera. Third, the feature points on the current key frame are projected onto the world coordinate system, and the reconstructed map is incrementally expanded. Finally, output the complete reconstructed map.
  • the embodiments of the present invention provide a method, a device, and a terminal device for reconstructing a scene, so as to solve the problem that the prior art is difficult to detect a sufficiently accurate feature matching relationship in a scene with low texture and high texture repeatability. It is difficult to get an accurate camera pose, which makes it difficult to accurately reconstruct the scene. question.
  • a first aspect of the embodiments of the present invention provides a method for three-dimensional reconstruction of a scenario, including:
  • Matching the second feature point with the first feature point determining a matching relationship between the second feature point set and the first feature point set, and matching the second feature plane with the first feature plane to determine a matching relationship between the second feature plane set and the first feature plane set;
  • a map is created according to all the markers including the maps of all the first feature points and all the first feature planes, and the three-dimensional reconstruction of the scene is realized.
  • a second aspect of the present invention provides a three-dimensional reconstruction apparatus for a scenario, including:
  • a color map and a depth map acquiring unit for acquiring a color map and a depth map of the scene
  • a first feature extraction unit configured to extract a first feature point according to a color map of the first frame, and construct a first feature point set
  • a point cloud map generating unit configured to generate a point cloud map according to the color map and the depth map, extract a first feature plane according to a point cloud image of the first frame, and construct a first feature plane set, the first feature point and the first The feature plane constitutes the initial marker of the map;
  • a second feature extraction unit configured to extract a second feature point from the color map of the current frame, construct a second feature point set, and extract a second point cloud from the point cloud image corresponding to the color map of the current frame a feature plane, constructing a second feature plane set;
  • a matching relationship determining unit configured to match the second feature point with the first feature point, determine a matching relationship between the second feature point set and the first feature point set, and set a second feature plane and a second feature plane Determining, by a feature plane, a matching relationship between the second feature plane set and the first feature plane set;
  • a key frame determining unit configured to: according to the matching relationship between the second feature point set and the first feature point set, the matching relationship between the second feature plane set and the first feature plane set, and a current frame and The number of interval frames of the last key frame determines whether the current frame is a key frame.
  • a flag adding unit of the map configured to project the second feature point and the second feature plane of the key frame of the effective initial camera pose onto the world coordinate system, and the projected second feature point as the newly added first feature point Adding to the first feature point set, and the projected second feature plane is added to the first feature plane set as a newly added first feature plane;
  • a scene reconstruction unit configured to create a map according to all the markers of the map including all the first feature points and all the first feature planes after obtaining the first feature points and the first feature planes of all the key frames in the color map, Realize the 3D reconstruction of the scene.
  • a third aspect of an embodiment of the present invention provides a terminal device including a memory, a processor, and a computer program stored in the memory and operable on the processor, when the processor executes the computer program The steps of implementing the three-dimensional reconstruction method of the above scenario.
  • a fourth aspect of the embodiments of the present invention provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the three-dimensional reconstruction method of the scene.
  • the embodiment of the present invention has the beneficial effects that since the current frame is used to calculate the pose of the current frame only when the current frame is a potential key frame, the calculation amount and the calculation amount can be greatly reduced.
  • the storage load is optimized only by the pose of the key frame and related markers, which significantly improves the efficiency of scene reconstruction and also extends the algorithm to the reconstruction of a wide range of indoor scenes.
  • Simultaneously Pose estimation using feature points and feature planes makes the algorithm accurate and robust to camera pose estimation and scene reconstruction in areas with low texture and texture repeatability.
  • FIG. 1 is a flowchart of a method for three-dimensional reconstruction of a scene according to an embodiment of the present invention
  • FIG. 2 is a schematic structural diagram of a three-dimensional reconstruction apparatus for a scene according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a terminal device according to an embodiment of the present invention.
  • Figure 1 shows a flow chart of a three-dimensional reconstruction method of a scene, which is detailed as follows:
  • a color map and a depth map corresponding to a scene requiring three-dimensional reconstruction are acquired from an RGB-D camera.
  • the default first frame is a key frame and the initial camera pose is set to a 4*4 unit matrix as a reference coordinate system of the map.
  • the first feature point is extracted from the first frame as a map flag.
  • the extracted one or more first feature points are used to construct a first set of feature points.
  • the ORB (Oriented FAST and Rotated BRIEF) descriptor can be detected and extracted from the color map by using the OpenCV tool, and the ORB descriptor is used to determine the first feature point, thereby obtaining the first feature point set.
  • Step S13 generating a point cloud image according to the color map and the depth map, extracting a first feature plane according to a point cloud image of the first frame, and constructing a first feature plane set, where the first feature point and the first feature plane form a map Initial marker
  • first feature point and the first feature plane extracted by the first frame of the color map and the point cloud map are only a part of all the markers that are ultimately used to create the map.
  • Step S14 extracting a second feature point from the color map of the current frame, constructing a second feature point set, and extracting a second feature plane from the point cloud corresponding to the color map of the current frame in the point cloud image, constructing the first Two feature plane sets;
  • the frame read according to the reading rule is used as the current frame of the camera, wherein the reading rule can be set to: the next frame read is at least the previous one read Frame interval K1 frame (This K1 is related to the frame rate of the RGB-D camera and the speed of the movement. In the experimental part of this system, K1 is set to 10 when the frame rate of the RGB-D camera is 30 Hz).
  • Step S15 matching the second feature point with the first feature point, determining a matching relationship between the second feature point set and the first feature point set, and matching the second feature plane with the first feature plane Determining a matching relationship between the second feature plane set and the first feature plane set;
  • the second feature points in the second feature point set are matched with the first feature points in the first feature point set to determine a matching relationship between the second feature point set and the first feature point set.
  • the matching relationship between the second feature point set and the first feature point set refers to matching by using a nearest neighbor algorithm feature descriptor in the OpenCV library to obtain a corresponding matching relationship, to determine whether the first feature point set is There is a first feature point that matches the second feature point.
  • the matching relationship between the second feature plane set and the first feature plane set is searched by the direction of violent matching, because the RGB-D camera angle of view ( ⁇ 120°) and the observation distance are limited (0.3m-5m), so the phase The number of feature planes of the neighbor key frames is limited. Therefore, the brute force matching method is used to compare all the potential planes at a time, and the matching relationship between the second feature plane set and the first feature plane set is determined.
  • Step S16 according to the matching relationship between the second feature point set and the first feature point set, the matching relationship between the second feature plane set and the first feature plane set, and the current frame and the previous key frame.
  • the number of interval frames determines whether the current frame is a key frame
  • step S16 specifically includes:
  • A1 Determine whether the number of interval frames of the current frame and the previous key frame is greater than a preset interval frame number threshold.
  • the preset interval frame number threshold is related to the frame rate of the RGB-D camera and the speed of the movement, for example, 10 when the frame rate of the RGB-D camera is 30 Hz.
  • the last key frame is the last inserted key frame, which refers to the second feature point (or the second feature plane) of the last projection as a new map mark. The corresponding frame.
  • the number of the first feature points matching the second feature point according to the first feature point set, and the first feature, when the interval frame number of the current frame and the previous key frame is greater than a preset interval frame number threshold Determining, by the plane, a number of first feature planes that match the second feature plane, determining whether the current frame is a key frame;
  • determining the matching relationship between the current frame and the map marker for example, determining whether the feature point and the feature plane satisfy a preset condition to determine whether the current frame is a key frame, for example, first calculating a value obtained by the following formula: the first feature The number of the first feature points matching the second feature point in the point set + the preset feature plane threshold value * the number of the first feature plane in the first feature plane set matching the second feature plane, and determining whether the value exceeds the pre-predetermined value Set the condition threshold, if yes, determine that the current frame is a key frame; otherwise, determine that the current frame is not a key frame.
  • the current frame is a non-key frame, the next frame is inserted, and step S11 is performed.
  • the preset interval frame number threshold may be set to 10
  • the preset feature plane threshold may be set to 10
  • the preset condition threshold may be set to 100 or the like.
  • Step S17 determining, when the current frame is a key frame, whether an initial camera pose of the current frame is valid
  • step S17 specifically includes:
  • the amount of change of the rotation matrix of the current frame and the rotation matrix of the previous key frame is less than a preset rotation matrix threshold. And determining, when the translation vector of the current frame and the translation vector of the previous key frame are smaller than a preset translation vector threshold, determining an initial camera pose of the current frame, otherwise determining the current frame The initial camera pose is invalid.
  • the B1 specifically includes:
  • Po and P1 respectively represent a first feature point set
  • the first feature plane set, Po′ and P1′ respectively represent a second feature point set and a second feature plane set on the current frame
  • R k represents a rotation matrix
  • t k represents a translation vector
  • n j represents a normal vector of the jth first feature plane in P1
  • n′ j represents a normal vector of the jth second feature plane in P1′
  • w j Represents the weight of the jth plane in P1
  • 2 " represents the 2-norm operation
  • the g2o is an open source optimization tool that is mainly used to solve some optimization problems. Among them, g2o refers to A general framework for graph optimization.
  • step S11 is performed.
  • the pose of the current frame is calculated by the matching relationship of the feature only when the current frame is a potential key frame, the calculation amount and the storage load can be greatly reduced, and only the pose of the key frame and related markers are used. Optimization, significantly improving the efficiency of scene reconstruction, and also extending the algorithm to the reconstruction of a wide range of indoor scenes.
  • Step S18 projecting the second feature point and the second feature plane of the key frame of the effective initial camera pose onto the world coordinate system, and adding the projected second feature point as the newly added first feature point to the first
  • the feature points are concentrated, and the second feature plane after the projection is added to the first feature plane set as a newly added first feature plane;
  • the first frame (the default is the key frame) and the initial camera pose is set to a 4*4 unit matrix as the reference coordinate system of the map.
  • the second feature point is projected onto the world coordinate system to expand the first feature point.
  • the number of first feature points of the set is
  • the word bag model is mainly used for accurate loop detection and relocation.
  • the system detects the loopback of each keyframe when it is inserted. If a loopback is detected, a relative camera pose estimate is calculated between the current frame and the loopback frame, and all the keys appearing on the loopback are eliminated in turn.
  • the cumulative error produced by the camera pose estimation of the frame is aligned at both ends of the loop and merged with the map points at both ends of the loop.
  • Step S19 after obtaining the first feature point and the first feature plane of all the key frames in the color map, creating a map according to all the markers including the maps of all the first feature points and all the first feature planes, implementing the scene Three-dimensional reconstruction.
  • the method includes:
  • the initial camera pose of all the key frames is subjected to global light speed adjustment to reduce the initial camera of all the key frames.
  • the cumulative error of the pose Specifically, due to the influence of the corresponding relationship between noise and error, there is a certain error in the effective initial camera pose estimated for each frame. When the error is accumulated, the entire map will have a serious error drift. Therefore, it is necessary to obtain the obtained error.
  • the effective initial camera pose for all keyframes is optimized.
  • a novel camera pose optimization method is proposed, that is, the global beam adjustment of the camera pose is performed by using the texture (feature point) and the geometric feature (feature plane) at the same time.
  • Construction optimization problem with Point sets and plane sets representing all markers on the map:
  • x i , y i , z i represent the coordinates of the first feature point on the map, respectively.
  • n j , d j , N j respectively represent the normal vector of the first feature plane, the distance between the first feature plane and the world coordinate system, and the number of feature points in the effective threshold range of the first feature plane.
  • the code implementation process does not strictly substitute the coordinates of some feature points into the plane expression, the value is equal to 0 as the point on the plane, but as long as the value is below a certain threshold. It is considered to be a point on the plane, and therefore, the range that meets the requirements is referred to as the first feature plane effective threshold range.
  • ⁇ (R 1 , t 1 ), (R 2 , t 2 ), ..., (R s , t s ) ⁇ respectively represent the initial estimates of the camera poses of all key frames, constructing a global beam adjustment
  • the optimization problem is as follows:
  • the variables used for optimization include all map markers (all first feature points and first feature faces), and camera poses for all key frames. Representing the ith second feature point in the second feature point set with the kth key frame, and p i representing the map Corresponding first feature point.
  • the reconstructed scene is globally optimized after the scene reconstruction, and the optimized scene is obtained, the estimation of the camera pose and the creation of the scene can be ensured in the region with low texture and texture repeatability. Accurate and robust.
  • a color map and a depth map of the scene are acquired, a first feature point is extracted according to the first frame of the color map, a first feature point set is constructed, and a point cloud image is generated according to the color map and the depth map. Extracting a first feature plane according to the first frame of the point cloud image, and constructing a first feature plane set, where the first feature point and the first feature plane constitute an initial marker of the map, and the second frame is extracted from the current frame.
  • Feature plane concentration after obtaining the first feature point and the first feature plane of all key frames in the color map, according to all first feature points and all first feature planes
  • All markers of the map create maps that enable 3D reconstruction of the scene. Since the pose of the current frame is calculated by using the matching relationship of the feature only when the current frame is a key frame, the calculation amount and the storage load can be greatly reduced, thereby improving the efficiency of scene reconstruction, and also facilitating the expansion to a wide range.
  • FIG. 2 is a schematic structural diagram of a three-dimensional reconstruction apparatus for a scene according to an embodiment of the present invention. For convenience of description, only parts related to the embodiment are shown.
  • the three-dimensional reconstruction apparatus of the scene includes: a color map and depth map acquisition unit 21, a first feature extraction unit 22, a point cloud map generation unit 23, a second feature extraction unit 24, a matching relationship determination unit 25, a key frame determination unit 26, and an initial The camera pose validity determination unit 27, the map addition unit 28 of the map, and the scene reconstruction unit 29. among them:
  • the color map and depth map obtaining unit 21 is configured to acquire a color map and a depth map of the scene.
  • the first feature extraction unit 22 is configured to extract a first feature point according to a color map of the first frame, and construct a first feature point set.
  • a point cloud map generating unit 23 configured to generate a point cloud map according to the color map and the depth map, extract a first feature plane according to a point cloud map of the first frame, and construct a first feature plane set, where the first feature point and the first feature point A feature plane constitutes the initial marker of the map.
  • a second feature extraction unit 24 configured to extract a second feature point from a color map of the current frame, construct a second feature point set, and extract a point cloud corresponding to the color map of the current frame from the point cloud image
  • the second feature plane constructs a second feature plane set.
  • the current frame is read according to a reading rule, wherein the reading rule may be set to: the next frame read is at least K1 frame from the previous frame read (the K1 and The frame rate of the RGB-D camera and the speed of the movement are related. When the frame rate of the RGB-D camera is 30 Hz, K1 is set to 10).
  • the matching relationship between the second feature point set and the first feature point set refers to matching by using a nearest neighbor algorithm feature descriptor in the OpenCV library to obtain a corresponding matching relationship, so as to determine whether the first feature point set exists.
  • the matching relationship between the second feature plane and the first feature plane is searched by the direction of violent matching, because the RGB-D camera angle of view ( ⁇ 120°) and the observation distance are limited (0.3m-5m), so the adjacent key
  • the number of feature planes of the frame is limited, so the brute force matching method is used to compare all the potential planes at a time, and the matching relationship between the second feature plane and the first feature plane is determined.
  • a key frame determining unit 26 configured to: according to the matching relationship between the second feature point set and the first feature point set, the matching relationship between the second feature plane set and the first feature plane set, and a current frame The interval frame number of the previous key frame is used to determine whether the current frame is a key frame.
  • the key frame determining unit 26 includes:
  • the interval frame number judging module is configured to determine whether the interval frame number of the current frame and the last key frame is greater than a preset interval frame number threshold.
  • the preset interval frame number threshold is related to the frame rate of the RGB-D camera and the speed of the movement.
  • determining whether the current frame is a key frame by determining whether the feature point and the feature plane meet the preset condition in the matching relationship between the current frame and the map marker, for example, determining that the first feature point set matches the second feature point.
  • the initial camera pose validity determining unit 27 is configured to determine whether the initial camera pose of the current frame is valid when the current frame is a key frame.
  • the initial camera pose validity determining unit 27 includes:
  • An initial camera pose estimation module configured to: when the current frame is a key frame, according to a matching relationship between the second feature point set and the first feature point set, the second feature plane set, and the A matching relationship of the first set of feature planes estimates an initial camera pose of the current frame, the initial camera pose of the current frame including a rotation matrix and a translation vector.
  • Po and P1 respectively represent a first feature point set
  • Po′ and P1′ respectively represent a second feature point set and a second feature plane set on the current frame
  • the flag adding unit 28 of the map is configured to project the second feature point and the second feature plane of the key frame of the effective initial camera pose onto the world coordinate system, and the projected second feature point is added as the first feature.
  • the point is added to the first feature point set, and the projected second feature plane is added to the first feature plane set as a newly added first feature plane.
  • the camera pose of the first frame (each frame of the RGB-D camera includes a color map and a corresponding depth map) can be used as the world coordinate system.
  • a scene reconstruction unit 29 configured to create a map according to all the markers including the maps of all the first feature points and all the first feature planes after obtaining the first feature points and the first feature planes of all the key frames in the color map. To achieve 3D reconstruction of the scene.
  • An optimization module configured to perform global light speed adjustment on the effective initial camera pose of all key frames after obtaining the first feature point and the first feature plane of all key frames in the color map, to reduce all the keys The cumulative error of the initial camera pose of the frame;
  • a three-dimensional reconstruction module configured to project a second feature point and a second feature plane of all key frames that cancel the cumulative error onto the world coordinate system, to obtain an optimized first feature point set and a new first feature of all key frames And acquiring a map according to the optimized first feature point set and the optimized first feature plane set to implement three-dimensional reconstruction of the scene.
  • FIG. 3 is a schematic diagram of a terminal device according to an embodiment of the present invention.
  • the terminal device 3 of this embodiment includes a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and operable on the processor 30.
  • the processor 30 executes the computer program 32, the steps in the embodiment of the three-dimensional reconstruction method of the above various scenarios are implemented, for example, steps S11 to S19 shown in FIG.
  • the processor 30 executes the computer program 32
  • the functions of the modules/units in the above various device embodiments are implemented, such as the functions of the modules 21 to 29 shown in FIG. 2.
  • the computer program 32 can be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 30 to complete this invention.
  • the one or more modules/units may be a series of computer program instruction segments capable of performing a particular function, the instruction segments being used to describe the execution of the computer program 32 in the terminal device 3.
  • the computer program 32 may be divided into a color map and depth map acquisition unit, a first feature extraction unit, a point cloud map generation unit, a second feature extraction unit, a matching relationship determination unit, a key frame determination unit, and an initial camera pose.
  • the validity judgment unit, the map addition unit and the scene reconstruction unit, and the specific functions of each module are as follows:
  • a color map and a depth map acquiring unit for acquiring a color map and a depth map of the scene
  • a first feature extraction unit configured to extract a first feature point according to a color map of the first frame, and construct a first Feature point set
  • a point cloud map generating unit configured to generate a point cloud map according to the color map and the depth map, extract a first feature plane according to a point cloud image of the first frame, and construct a first feature plane set, the first feature point and the first The feature plane constitutes the initial marker of the map;
  • a second feature extraction unit configured to extract a second feature point from the color map of the current frame, construct a second feature point set, and extract a second point cloud from the point cloud image corresponding to the color map of the current frame a feature plane, constructing a second feature plane set;
  • a matching relationship determining unit configured to match the second feature point with the first feature point, determine a matching relationship between the second feature point set and the first feature point set, and set a second feature plane and a second feature plane Determining, by a feature plane, a matching relationship between the second feature plane set and the first feature plane set;
  • a key frame determining unit configured to: according to the matching relationship between the second feature point set and the first feature point set, the matching relationship between the second feature plane set and the first feature plane set, and a current frame and The number of interval frames of the last key frame determines whether the current frame is a key frame.
  • An initial camera pose validity determining unit configured to determine whether an initial camera pose of the current frame is valid when the current frame is a key frame
  • a flag adding unit of the map configured to project the second feature point and the second feature plane of the key frame of the effective initial camera pose onto the world coordinate system, and the projected second feature point as the newly added first feature point Adding to the first feature point set, and the projected second feature plane is added to the first feature plane set as a newly added first feature plane;
  • a scene reconstruction unit configured to create a map according to all the markers of the map including all the first feature points and all the first feature planes after obtaining the first feature points and the first feature planes of all the key frames in the color map, Realize the 3D reconstruction of the scene.
  • the terminal device 3 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server.
  • the terminal device 3 may include, but is not limited to, a processor 30 and a memory 31. It will be understood by those skilled in the art that FIG. 3 is merely an example of the terminal device 3, does not constitute a limitation of the terminal device 3, may include more or less components than those illustrated, or combine some components, or different components.
  • the terminal device may further include an input/output device, a network access device, a bus, and the like.
  • the processor 30 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc.
  • the general purpose processor may be a microprocessor or the processor or any conventional processor or the like.
  • the memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or a memory of the terminal device 3.
  • the memory 31 may also be an external storage device of the terminal device 3, for example, a plug-in hard disk provided on the terminal device 3, a smart memory card (SMC), and a secure digital (SD). Card, flash card, etc. Further, the memory 31 may also include both an internal storage unit of the terminal device 3 and an external storage device.
  • the memory 31 is used to store the computer program and other programs and data required by the terminal device 3.
  • the memory 31 can also be used to temporarily store data that has been output or is about to be output.
  • each functional unit and module described above is exemplified. In practical applications, the above functions may be assigned to different functional units as needed.
  • the module is completed by dividing the internal structure of the device into different functional units or modules to perform all or part of the functions described above.
  • Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the integrated unit may be hardware.
  • Formal implementation can also be implemented in the form of software functional units.
  • the specific names of the respective functional units and modules are only for the purpose of facilitating mutual differentiation, and are not intended to limit the scope of protection of the present application.
  • For the specific working process of the unit and the module in the foregoing system reference may be made to the corresponding process in the foregoing method embodiment, and details are not described herein again.
  • the disclosed apparatus/terminal device and method may be implemented in other manners.
  • the device/terminal device embodiments described above are merely illustrative
  • the division of the module or unit is only a logical function division, and the actual implementation may have another division manner, for example, multiple units or components may be combined or integrated into another system, or some Features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated modules/units if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the foregoing embodiments, and may also be completed by a computer program to instruct related hardware.
  • the computer program may be stored in a computer readable storage medium. The steps of the various method embodiments described above may be implemented when the program is executed by the processor. .
  • the computer program comprises computer program code, which may be in the form of source code, object code form, executable file or some intermediate form.
  • the computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM). , random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, computer readable media Does not include electrical carrier signals and telecommunication signals.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

场景的三维重建方法、装置及终端设备，包括：构建第一特征点集和第一特征平面集，第一特征点和第一特征平面组成地图的初始标志物；从当前帧的彩色图构建第二特征点集，从点云图中对应的点云构建第二特征平面集；确定第二特征点集与第一特征点集的匹配关系和第二特征平面集与第一特征平面集的匹配关系；根据确定的匹配关系以及当前帧与上一个关键帧的间隔帧数判断当前帧是否为关键帧；若是，判断当前帧的初始相机位姿是否有效；将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，得到新增的第一特征点和第一特征平面；根据地图的所有标志物创建地图，实现场景的三维重建。通过上述方法，使得重建的场景更准确。

Description

场景的三维重建方法、装置及终端设备 技术领域
本发明属于三维场景重建技术领域，尤其涉及场景的三维重建方法、装置及终端设备。
背景技术
由于场景的三维重建方法有助于很多实际应用，比如虚拟现实和增强现实，机器人的定位和路径规划，以及自动引导运输车的室内工作等等，因此受到研究人员越来越多的关注。利用RGB-D相机进行三维重建相比于双目相机具有独特的优势，因为RGB-D相机能够提供图像像素坐标对应的深度值，不用花费大量计算资源用于像素视差的计算。传统的基于RGB-D相机的三维重建方法主要是利用特征点进行特征的检测与匹配。
一般来说，基于RGB-D相机的三维重建主要是根据相机在多个角度观察到的场景，利用特征的空间位置关系计算相机的相机位姿从而恢复场景的三维信息。基于RGB-D相机的三维重建算法主要分为以下几个步骤：第一，从获取的图像中进行特征点的检测和提取，并且利用对应的深度图获取相应特征点的空间坐标。第二，获取的每帧都进行特征点的检测和提取，并将检测到的特征点与地图上的点进行特征的匹配，计算当前相机的相机位姿。第三，将当前的关键帧上的特征点投影到世界坐标系上，对重建的地图进行增量的扩充。最后，输出完整的重建地图。
广泛的研究表明大部分基于特征点进行特征检测和匹配的三维重建算法在低纹理和纹理重复性很强的场景内很难检测到足够准确的特征匹配关系，因而很难得到一个准确的相机位姿。
发明内容
有鉴于此，本发明实施例提供了场景的三维重建方法、装置及终端设备，以解决现有技术在低纹理和纹理重复性很强的场景内很难检测到足够准确的特征匹配关系，因而很难得到一个准确的相机位姿，从而导致场景难以准确重建的问 题。
本发明实施例的第一方面提供了一种场景的三维重建方法，包括：
获取场景的彩色图和深度图；
根据第一帧的彩色图提取第一特征点，构建第一特征点集；
根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物；
从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集；
将所述第二特征点与所述第一特征点匹配，确定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系；
根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧；
在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效；
将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中；
在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
本发明实施例的第二方面提供了一种场景的三维重建装置，包括：
彩色图和深度图获取单元，用于获取场景的彩色图和深度图；
第一特征提取单元，用于根据第一帧的彩色图提取第一特征点，构建第一特征点集；
点云图生成单元，用于根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一 特征平面组成地图的初始标志物；
第二特征提取单元，用于从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集；
匹配关系确定单元，用于将所述第二特征点与所述第一特征点匹配，确定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系；
关键帧判断单元，用于根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧。
初始相机位姿有效性判断单元，用于在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效；
地图的标志新增单元，用于将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中；
场景重建单元，用于在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
本发明实施例的第三方面提供了一种终端设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述场景的三维重建方法的步骤。
本发明实施例的第四方面提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述场景的三维重建方法的步骤。
本发明实施例与现有技术相比存在的有益效果是：由于只有在当前帧为潜在的关键帧时，才利用特征的匹配关系计算当前帧的位姿，因此，能够极大减少计算量和存储负载，仅利用关键帧的位姿和相关的标志物进行优化，显著地提高了场景重建的效率，也有利于将算法扩展到对大范围室内场景的重建。此外同时利 用特征点和特征平面进行位姿估计，使得算法在低纹理和纹理重复性很强的区域，也能保证对相机位姿估计以及场景重建的准确性和高鲁棒性。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例提供的一种场景的三维重建装置方法的流程图；
图2是本发明实施例提供的一种场景的三维重建装置的结构示意图；
图3是本发明实施例提供的终端设备的示意图。
具体实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本发明实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本发明。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本发明的描述。
为了说明本发明所述的技术方案，下面通过具体实施例来进行说明。
图1示出了一种场景的三维重建方法的流程图，详述如下：
步骤S11，获取场景的彩色图和深度图；
具体地，从RGB-D相机中获取需要三维重建的场景对应的彩色图和深度图。
步骤S12，根据第一帧的彩色图提取第一特征点，构建第一特征点集；
具体地，默认第一帧为关键帧且初始的相机位姿设为一个4*4的单位矩阵作为地图的参考坐标系，此时，从该第一帧中提取第一特征点作为地图的标志物的一个分量，提取的一个或多个第一特征点用于构建第一特征点集。具体地，可以利用OpenCV工具从彩色图中检测并提取ORB(Oriented FAST and Rotated BRIEF)描述符，该ORB描述符用于确定第一特征点，进而得到第一特征点集。
步骤S13，根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物；
需要指出的是，由彩色图和点云图的第一帧提取的第一特征点和第一特征平面只为最终用于创建地图的所有标志物的一部分。
步骤S14，从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集；
具体地，从RGB-D相机的第一帧开始，根据读取规则读取的帧作为相机的当前帧，其中，读取规则可设为：读取的下一帧至少与读取的上一帧间隔K1帧(该K1与RGB-D相机的帧率以及移动的速度有关，本系统实验环节当RGB-D相机的帧率为30Hz时K1设置为10)。
步骤S15，将所述第二特征点与所述第一特征点匹配，确定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系；
具体地，将第二特征点集中的第二特征点，与，第一特征点集中的第一特征点匹配，以确定第二特征点集与第一特征点集的匹配关系。
其中，这里的第二特征点集与所述第一特征点集的匹配关系，是指利用OpenCV库里面的最近邻算法特征描述符进行匹配得到对应的匹配关系，以判断第一特征点集是否存在与第二特征点匹配的第一特征点。第二特征平面集与所述第一特征平面集的匹配关系是通过暴力匹配的方向进行查找，因为RGB-D相机视角(<120°)和观测距离的有限(0.3m-5m)，所以相邻关键帧的特征平面数量有限，故采用暴力匹配的方法一次比较潜在的所有平面，确定所述第二特征平面集与所述第一特征平面集的匹配关系。
步骤S16，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧；
可选地，所述步骤S16具体包括：
A1、判断当前帧与上一个关键帧的间隔帧数是否大于预设的间隔帧数阈值。其中，预设的间隔帧数阈值与RGB-D相机的帧率以及移动的速度有关，例如，当RGB-D相机的帧率为30Hz时设置为10。其中，上一个关键帧为上一次插入的关键帧，是指上一次投影的第二特征点(或第二特征平面)作为新的地图标志 物对应的帧。
A2、在当前帧与上一个关键帧的间隔帧数大于预设的间隔帧数阈值时，根据第一特征点集中与第二特征点匹配的第一特征点的个数，和，第一特征平面集中与第二特征平面匹配的第一特征平面的个数，判断所述当前帧是否为关键帧；
具体地，通过判断当前帧与地图标志物的匹配关系，如判断特征点和特征平面是否满足预设的条件来判断当前帧是否为关键帧，例如，首先计算以下公式得到的值：第一特征点集中与第二特征点匹配的第一特征点的个数+预设特征平面阈值*第一特征平面集中与第二特征平面匹配的第一特征平面的个数，再判断该值是否超过预设的条件阈值，若是，则判定当前帧为关键帧，否则，判定当前帧不为关键帧。当然，若当前帧为非关键帧，则插入下一帧，并执行步骤S11。
例如，预设的间隔帧数阈值可设为10，而预设特征平面阈值可设为10，预设的条件阈值可设为100等。
步骤S17，在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效；
可选地，所述步骤S17具体包括：
B1、在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系预估所述当前帧的初始相机位姿，所述当前帧的初始相机位姿包括旋转矩阵和平移向量；
B2、将所述当前帧的初始相机位姿与上一个关键帧的初始相机位姿比较，在所述当前帧的旋转矩阵与上一个关键帧的旋转矩阵的改变量小于预设的旋转矩阵阈值，且在所述当前帧的平移向量与上一个关键帧的平移向量的改变量小于预设的平移向量阈值时，判定所述当前帧的初始相机位姿有效，否则，判定所述当前帧的初始相机位姿无效。
可选地，所述B1具体包括：
假设
Figure PCTCN2017085824-appb-000001
其中，Po和Pl分别表示第一特征点集，第一特征平面集，Po'和Pl'分别表示当前帧上第二特征点集和第二特征平面集；
在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系确定下式：
Figure PCTCN2017085824-appb-000002
其中，Rk表示旋转矩阵，tk表示平移向量，nj表示Pl中第j个第一特征平面的法向量，n'j表示Pl'中第j个第二特征平面的法向量，wj表示Pl中第j个平面的权重，“||||2”表示2-范数运算；对上式进行优化求解预估为关键帧的当前帧的初始相机位姿，如通过g2o进行优化求解，该g2o是一种开源的优化工具，主要用来解决一些优化问题的工具库，其中，g2o是指A general framework for graph optimization。
当然，若当前帧的相机位姿无效，则插入下一帧，并执行步骤S11。
由于只有在当前帧为潜在的关键帧时，才利用特征的匹配关系计算当前帧的位姿，因此，能够极大减少计算量和存储负载，仅利用关键帧的位姿和相关的标志物进行优化，显著地提高了场景重建的效率，也有利于将算法扩展到对大范围室内场景的重建。
步骤S18，将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中；
其中，将第一帧(默认为关键帧)且初始的相机位姿设为一个4*4的单位矩阵作为地图的参考坐标系。
该步骤中，当第一特征点集不存在与从当前关键帧提取的第二特征点相同的第一特征点时，将该第二特征点投影到世界坐标系上，以扩大第一特征点集的第一特征点的数量。
需要指出的是，当初始相机位姿没有预估出来时，表明特征点(或特征平面)的匹配不准确或者不够造成追踪丢失，此时，需要进行重定位，重新提取特征点(或特征平面)，以实现特征点(或特征平面)的重定位。其中，循环检测(Loopclosure)主要用于检测当前的场景是否在地图上被描述过。循环闭合的重要性体现在能够有效的消除累计的漂移，形成一个全局一致的重建地图。重定位是指当相机遭遇阻塞或者纯旋转问题发生特征追踪丢失的情况下，进行自我的定位从而恢复运动状态的过程。本发明提出的室内场景三维重建的方法，内部嵌入 了一个词袋模型的模块，具体的实现过程可以参考：Gálvez-López D,Tardos J D.Bags of binary words for fast place recognition in image sequences[J].IEEE Transactions on Robotics,2012,28(5):1188-1197。词袋模型主要用于准确的回环检测和重定位。系统在插入每个关键帧的时候对其进行回环的检测，如果被检测到一个回环，则对当前帧和回环的帧之间计算一个相对的相机位姿估计，依次消除回环上出现的所有关键帧的相机位姿估计产生的累计误差，对齐回环的两端，并融合回环两端的地图点。
步骤S19，在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
可选地，在所述步骤S19，包括：
B1、在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，将所有关键帧的初始相机位姿进行全局光速法平差，以减少所述所有关键帧的初始相机位姿的累计误差。具体地，由于噪声和错误的对应关系的影响，每一帧估计的有效初始相机位姿都会存在一定的误差，当误差累计下去，整个地图就会发生严重的误差漂移，因此，需要对得到的所有关键帧的有效初始相机位姿进行优化。
在本发明实施例中，提出一种新颖的相机位姿优化的方法，即同时利用纹理(特征点)与几何特征(特征平面)进行相机位姿的全局光束法平差。构造优化问题：
Figure PCTCN2017085824-appb-000003
和
Figure PCTCN2017085824-appb-000004
分别表示地图上所有标志物的点集和平面集：
Figure PCTCN2017085824-appb-000005
其中：pi＝(xi,yi,zi)T以及πj＝(nj,dj,Nj)分别表示地图上的特征点和特征平面。xi,yi,zi分别表示地图上第一特征点的坐标。nj,dj,Nj分别表示第一特征平面的法向量、第一特征平面与世界坐标系的距离，以及第一特征平面有效阈值范围内的特征点的个数。需要指出的是，由于代码实现的过程中并不是严格的将某些特征点的坐标代入到平面表达式之后的值等于0才作为平面上的点，而是只要这个值低于某个阈值就认为是平面上的点，因此，将符合要求的范围称为第一特征平面有效阈值范围。{(R1,t1),(R2,t2),...,(Rs,ts)}分别表示所有关键帧的相机位姿的初始估计， 构造全局的光束法平差的优化问题如下：
Figure PCTCN2017085824-appb-000006
其中：用于优化的变量包括所有的地图标志物(所有第一特征点和第一特征面)，以及所有的关键帧的相机位姿。
Figure PCTCN2017085824-appb-000007
表示与第k个关键帧的第二特征点集中的第i个第二特征点，而pi表示地图上与其
Figure PCTCN2017085824-appb-000008
对应的第一特征点。
B2、将消除累计误差的所有关键帧的初始相机位姿的第二特征点和第二特征平面投影到世界坐标系上，得到所有彩色图中所有关键帧的新的第一特征点集和新的第一特征平面集，根据所述新的第一特征点集和所述新的第一特征平面集创建地图，实现场景的三维重建。
由于在场景重建之后还对重建的场景进行全局优化，进而得到优化后的场景，因此，使得在低纹理和纹理重复性很强的区域，也能保证对相机位姿的估计以及场景的创建更加准确，且鲁棒性更强。
本发明实施例中，获取场景的彩色图和深度图，根据所述彩色图的第一帧提取第一特征点，构建第一特征点集，根据所述彩色图和所述深度图生成点云图，根据所述点云图的第一帧提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物，从所述的当前帧提取第二特征点，构建第二特征点集，以及，从所述当前帧的点云图中提取第二特征平面，构建第二特征平面集，将所述第二特征点与所述第一特征点匹配，确定所述第二特征点与所述第一特征点的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面与所述第一特征平面的匹配关系，根据所述第二特征点与所述第一特征点的匹配关系、所述第二特征平面与所述第二特征平面的匹配关系以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧，在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效，将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中，在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面 的地图的所有标志物创建地图，实现场景的三维重建。由于只有在当前帧为关键帧时，才利用特征的匹配关系计算当前帧的位姿，因此，能够极大减少计算量和存储负载，从而提高场景重建的效率，也有利于扩展到对大范围室内场景的重建，并且，在场景重建之后还对重建的场景进行全局优化，进而得到优化后的场景，因此，使得在低纹理和纹理重复性很强的区域，也能保证对相机位姿的估计以及场景重建建更加准确和鲁棒。
应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本发明实施例的实施过程构成任何限定。
图2示出了本发明实施例提供的一种场景的三维重建装置的结构示意图，为了便于说明，仅示出了与本实施例相关的部分。
该场景的三维重建装置包括：彩色图和深度图获取单元21、第一特征提取单元22、点云图生成单元23、第二特征提取单元24、匹配关系确定单元25、关键帧判断单元26、初始相机位姿有效性判断单元27、地图的标志新增单元28、场景重建单元29。其中：
彩色图和深度图获取单元21，用于获取场景的彩色图和深度图。
第一特征提取单元22，用于根据第一帧的彩色图提取第一特征点，构建第一特征点集。
具体地，默认彩色图中的第一帧为该彩色图的关键帧。
点云图生成单元23，用于根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物。
第二特征提取单元24，用于从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集。
具体地，从相机的第一帧开始，根据读取规则读取当前帧，其中，读取规则可设为：读取的下一帧至少与读取的上一帧间隔K1帧(该K1与RGB-D相机的帧率以及移动的速度有关，当RGB-D相机的帧率为30Hz时K1设置为10)。
匹配关系确定单元25，用于将所述第二特征点与所述第一特征点匹配，确 定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系。
其中，这里的第二特征点集与所述第一特征点集的匹配关系是指利用OpenCV库里面的最近邻算法特征描述符进行匹配得到对应的匹配关系，以判断第一特征点集是否存在与第二特征点匹配的第一特征点。第二特征平面与所述第一特征平面的匹配关系是通过暴力匹配的方向进行查找，因为RGB-D相机视角(<120°)和观测距离的有限(0.3m-5m),所以相邻关键帧的特征平面数量有限，故采用暴力匹配的方法一次比较潜在的所有平面，确定所述第二特征平面与所述第一特征平面的匹配关系。
关键帧判断单元26，用于根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧。
可选地，所述关键帧判断单元26包括：
间隔帧数判断模块，用于判断当前帧与上一个关键帧的间隔帧数是否大于预设的间隔帧数阈值。
其中，预设的间隔帧数阈值与RGB-D相机的帧率以及移动的速度有关。
特征点个数判断模块，用于在当前帧与上一个关键帧的间隔帧数大于预设的间隔帧数阈值时，根据第一特征点集中与第二特征点匹配的第一特征点的个数，和，第一特征平面集中与第二特征平面匹配的第一特征平面的个数，判断所述当前帧是否为关键帧。
具体地，通过判断当前帧与地图标志物的匹配关系中特征点和特征平面是否满足预设的条件来判断当前帧是否为关键帧，例如，判断第一特征点集中与第二特征点匹配的第一特征点的个数+预设特征平面阈值*第一特征平面集中与第二特征平面匹配的第一特征平面的个数是否超过预设的条件阈值，若是，则判定当前帧为关键帧，否则，判定当前帧不为关键帧。
初始相机位姿有效性判断单元27，用于在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效。
可选地，所述初始相机位姿有效性判断单元27，包括：
初始相机位姿预估模块，用于在所述当前帧为关键帧时，根据所述第二特 征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系预估所述当前帧的初始相机位姿，所述当前帧的初始相机位姿包括旋转矩阵和平移向量。具体地，假设
Figure PCTCN2017085824-appb-000009
其中，Po和Pl分别表示第一特征点集，第一特征平面集，Po'和Pl'分别表示当前帧上第二特征点集和第二特征平面集；
在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系确定下式：
Figure PCTCN2017085824-appb-000010
其中，Rk表示旋转矩阵，tk表示平移向量，nj表示Pl中第j个第一特征平面的法向量，n'j表示Pl'中第j个第二特征平面的法向量，wj表示Pl中第j个平面的权重，“||||2”表示2-范数运算；对上式进行优化求解预估为关键帧的当前帧的初始相机位姿。
初始相机位姿比较模块，用于将所述当前帧的初始相机位姿与上一个关键帧的初始相机位姿比较，在所述当前帧的旋转矩阵与上一个关键帧的旋转矩阵的改变量小于预设的旋转矩阵阈值，且在所述当前帧的平移向量与上一个关键帧的平移向量的改变量小于预设的平移向量阈值时，判定所述当前帧的初始相机位姿有效，否则，判定所述当前帧的初始相机位姿无效。
地图的标志新增单元28，用于将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中。
其中，可将第一帧(RGB-D相机的每一帧就包括一张彩色图和对应的深度图)的相机位姿作为世界坐标系。
需要指出的是，当初始相机位姿没有预估出来时，表明特征点(或特征平面)追踪丢失，此时，需要重新提取特征点(或特征平面)，以实现特征点(或特征平面)的重定位。
场景重建单元29，用于在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
可选地，所述场景重建单元29，包括：
优化模块，用于在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，将所有关键帧的有效初始相机位姿进行全局光速法平差，以减少所述所有关键帧的初始相机位姿的累计误差；
三维重建模块，用于将消除累计误差的所有关键帧的第二特征点和第二特征平面投影到世界坐标系上，得到所有关键帧的优化后的第一特征点集和新的第一特征平面集，根据所述优化后的第一特征点集和所述优化后的第一特征平面集创建地图，实现场景的三维重建。
其中，优化模块和三维重建模块的具体实现过程在上面已描述，此处不再赘述。
图3是本发明一实施例提供的一种终端设备的示意图。如图3所示，该实施例的终端设备3包括：处理器30、存储器31以及存储在所述存储器31中并可在所述处理器30上运行的计算机程序32。所述处理器30执行所述计算机程序32时实现上述各个场景的三维重建方法实施例中的步骤，例如图1所示的步骤S11至S19。或者，所述处理器30执行所述计算机程序32时实现上述各装置实施例中各模块/单元的功能，例如图2所示模块21至29功能。
示例性的，所述计算机程序32可以被分割成一个或多个模块/单元，所述一个或者多个模块/单元被存储在所述存储器31中，并由所述处理器30执行，以完成本发明。所述一个或多个模块/单元可以是能够完成特定功能的一系列计算机程序指令段，该指令段用于描述所述计算机程序32在所述终端设备3中的执行过程。例如，所述计算机程序32可以被分割为彩色图和深度图获取单元、第一特征提取单元、点云图生成单元、第二特征提取单元、匹配关系确定单元、关键帧判断单元、初始相机位姿有效性判断单元、地图的标志新增单元、场景重建单元，各模块具体功能如下：
彩色图和深度图获取单元，用于获取场景的彩色图和深度图；
第一特征提取单元，用于根据第一帧的彩色图提取第一特征点，构建第一 特征点集；
点云图生成单元，用于根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物；
第二特征提取单元，用于从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集；
匹配关系确定单元，用于将所述第二特征点与所述第一特征点匹配，确定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系；
关键帧判断单元，用于根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧。
初始相机位姿有效性判断单元，用于在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效；
地图的标志新增单元，用于将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中；
场景重建单元，用于在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
所述终端设备3可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。所述终端设备3可包括但不仅限于，处理器30、存储器31。本领域技术人员可以理解，图3仅仅是终端设备3的示例，并不构成对终端设备3的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件，例如所述终端设备还可以包括输入输出设备、网络接入设备、总线等。
所称处理器30可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、 专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
所述存储器31可以是所述终端设备3的内部存储单元，例如终端设备3的硬盘或内存。所述存储器31也可以是所述终端设备3的外部存储设备，例如所述终端设备3上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，所述存储器31还可以既包括所述终端设备3的内部存储单元也包括外部存储设备。所述存储器31用于存储所述计算机程序以及所述终端设备3所需的其他程序和数据。所述存储器31还可以用于暂时地存储已经输出或者将要输出的数据。
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能单元、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能单元、模块完成，即将所述装置的内部结构划分成不同的功能单元或模块，以完成以上描述的全部或者部分功能。实施例中的各功能单元、模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中，上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。另外，各功能单元、模块的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。上述系统中单元、模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。
在本发明所提供的实施例中，应该理解到，所揭露的装置/终端设备和方法，可以通过其它的方式实现。例如，以上所描述的装置/终端设备实施例仅仅是示 意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通讯连接可以是通过一些接口，装置或单元的间接耦合或通讯连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的模块/单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明实现上述实施例方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、电载波信号、电信信号以及软件分发介质等。需要说明的是，所述计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括电载波信号和电信信号。
以上所述实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行 等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种场景的三维重建方法，其特征在于，包括：
    获取场景的彩色图和深度图；
    根据第一帧的彩色图提取第一特征点，构建第一特征点集；
    根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物；
    从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集；
    将所述第二特征点与所述第一特征点匹配，确定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系；
    根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧；
    在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效；
    将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中；
    在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
  2. 如权利要求1所述的场景的三维重建方法，其特征在于，所述根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧，具体包括：
    判断当前帧与上一个关键帧的间隔帧数是否大于预设的间隔帧数阈值；
    在当前帧与上一个关键帧的间隔帧数大于预设的间隔帧数阈值时，根据第一特征点集中与第二特征点匹配的第一特征点的个数，和，第一特征平面集中与第二特征平面匹配的第一特征平面的个数，判断所述当前帧是否为关键帧。
  3. 如权利要求1所述的场景的三维重建方法，其特征在于，所述在所述当前帧为关键帧时，判断为关键帧的当前帧的初始相机位姿是否有效，具体包括：
    在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系预估所述当前帧的初始相机位姿，所述当前帧的初始相机位姿包括旋转矩阵和平移向量；
    将所述当前帧的初始相机位姿与上一个关键帧的初始相机位姿比较，在所述当前帧的旋转矩阵与上一个关键帧的旋转矩阵的改变量小于预设的旋转矩阵阈值，且在所述当前帧的平移向量与上一个关键帧的平移向量的改变量小于预设的平移向量阈值时，判定所述当前帧的初始相机位姿有效，否则，判定所述当前帧的初始相机位姿无效。
  4. 如权利要求3所述的场景的三维重建方法，其特征在于，在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系预估所述当前帧的初始相机位姿，具体包括：
    假设
    Figure PCTCN2017085824-appb-100001
    其中，Po和Pl分别表示第一特征点集，第一特征平面集，Po'和Pl'分别表示当前帧上第二特征点集和第二特征平面集；
    在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系确定下式：
    Figure PCTCN2017085824-appb-100002
    其中，Rk表示旋转矩阵，tk表示平移向量，nj表示Pl中第j个第一特征平面的法向量，n'j表示Pl'中第j个第二特征平面的法向量，wj表示Pl中第j个平面的权重，“|| ||2”表示2-范数运算；对上式进行优化求解预估为关键帧的当前帧的初始相机位姿。
  5. 如权利要求1至4任一项所述的场景的三维重建方法，其特征在于，所述在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建包括：
    在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，将所有关键帧的有效初始相机位姿进行全局光速法平差，以减少所述所有关键帧的初始相机位姿的累计误差；
    将消除累计误差的所有关键帧的第二特征点和第二特征平面投影到世界坐标系上，得到所有关键帧的优化后的第一特征点集和新的第一特征平面集，根据所述优化后的第一特征点集和所述优化后的第一特征平面集创建地图，实现场景的三维重建。
  6. 一种场景的三维重建装置，其特征在于，包括：
    彩色图和深度图获取单元，用于获取场景的彩色图和深度图；
    第一特征提取单元，用于根据第一帧的彩色图提取第一特征点，构建第一特征点集；
    点云图生成单元，用于根据所述彩色图和所述深度图生成点云图，根据第一帧的点云图提取第一特征平面，构建第一特征平面集，所述第一特征点和第一特征平面组成地图的初始标志物；
    第二特征提取单元，用于从当前帧的彩色图提取第二特征点，构建第二特征点集，以及，从所述点云图中与所述当前帧的彩色图对应的点云提取第二特征平面，构建第二特征平面集；
    匹配关系确定单元，用于将所述第二特征点与所述第一特征点匹配，确定所述第二特征点集与所述第一特征点集的匹配关系，将第二特征平面与第一特征平面匹配，确定所述第二特征平面集与所述第一特征平面集的匹配关系；
    关键帧判断单元，用于根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系、以及当前帧与上一个关键帧的间隔帧数判断所述当前帧是否为关键帧。
    初始相机位姿有效性判断单元，用于在所述当前帧为关键帧时，判断所述当前帧的初始相机位姿是否有效；
    地图的标志新增单元，用于将有效初始相机位姿的关键帧的第二特征点和第二特征平面投影到世界坐标系上，投影后的第二特征点作为新增的第一特征点加入到所述第一特征点集中，投影后的第二特征平面作为新增的第一特征平面加入到所述第一特征平面集中；
    场景重建单元，用于在得到所述彩色图中所有关键帧的第一特征点和第一特征平面后，根据包括所有第一特征点和所有第一特征平面的地图的所有标志物创建地图，实现场景的三维重建。
  7. 如权利要求6所述的场景的三维重建装置，其特征在于，所述关键帧判断单元包括：
    间隔帧数判断模块，用于判断当前帧与上一个关键帧的间隔帧数是否大于预设的间隔帧数阈值；
    特征点个数判断模块，用于在当前帧与上一个关键帧的间隔帧数大于预设的间隔帧数阈值时，根据第一特征点集中与第二特征点匹配的第一特征点的个数，和，第一特征平面集中与第二特征平面匹配的第一特征平面的个数，判断所述当前帧是否为关键帧。
  8. 如权利要求6所述的场景的三维重建装置，其特征在于，所述初始相机位姿有效性判断单元，包括：
    初始相机位姿预估模块，用于在所述当前帧为关键帧时，根据所述第二特征点集与所述第一特征点集的匹配关系、所述第二特征平面集与所述第一特征平面集的匹配关系预估所述当前帧的初始相机位姿，所述当前帧的初始相机位姿包括旋转矩阵和平移向量；
    初始相机位姿比较模块，用于将所述当前帧的初始相机位姿与上一个关键帧的初始相机位姿比较，在所述当前帧的旋转矩阵与上一个关键帧的旋转矩阵的改变量小于预设的旋转矩阵阈值，且在所述当前帧的平移向量与上一个关键帧的平移向量的改变量小于预设的平移向量阈值时，判定所述当前帧的初始相机位姿有效，否则，判定所述当前帧的初始相机位姿无效。
  9. 一种终端设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1至5任一项所述方法的步骤。
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至5任一项所述方法的步骤。
PCT/CN2017/085824 2017-05-25 2017-05-25 场景的三维重建方法、装置及终端设备 WO2018214086A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/085824 WO2018214086A1 (zh) 2017-05-25 2017-05-25 场景的三维重建方法、装置及终端设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/085824 WO2018214086A1 (zh) 2017-05-25 2017-05-25 场景的三维重建方法、装置及终端设备

Publications (1)

Publication Number Publication Date
WO2018214086A1 true WO2018214086A1 (zh) 2018-11-29

Family

ID=64395123

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/085824 WO2018214086A1 (zh) 2017-05-25 2017-05-25 场景的三维重建方法、装置及终端设备

Country Status (1)

Country Link
WO (1) WO2018214086A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110162098A (zh) * 2019-07-03 2019-08-23 安徽理工大学 一种矿用无人机
GB2571378A (en) * 2018-02-23 2019-08-28 Sony Interactive Entertainment Europe Ltd Apparatus and method of mapping a virtual environment
CN110316376A (zh) * 2019-07-03 2019-10-11 安徽理工大学 一种用于探测矿井火灾的无人机
US11024054B2 (en) 2019-05-16 2021-06-01 Here Global B.V. Method, apparatus, and system for estimating the quality of camera pose data using ground control points of known quality
CN113223086A (zh) * 2021-06-09 2021-08-06 司法鉴定科学研究院 适用低质量监控视频的车辆运行状态的重建方法和系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103247075A (zh) * 2013-05-13 2013-08-14 北京工业大学 基于变分机制的室内环境三维重建方法
US20150254857A1 (en) * 2014-03-10 2015-09-10 Sony Corporation Image processing system with registration mechanism and method of operation thereof
CN105678842A (zh) * 2016-01-11 2016-06-15 湖南拓视觉信息技术有限公司 室内环境三维地图制作方法及装置
CN105913489A (zh) * 2016-04-19 2016-08-31 东北大学 一种利用平面特征的室内三维场景重构方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103247075A (zh) * 2013-05-13 2013-08-14 北京工业大学 基于变分机制的室内环境三维重建方法
US20150254857A1 (en) * 2014-03-10 2015-09-10 Sony Corporation Image processing system with registration mechanism and method of operation thereof
CN105678842A (zh) * 2016-01-11 2016-06-15 湖南拓视觉信息技术有限公司 室内环境三维地图制作方法及装置
CN105913489A (zh) * 2016-04-19 2016-08-31 东北大学 一种利用平面特征的室内三维场景重构方法

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2571378A (en) * 2018-02-23 2019-08-28 Sony Interactive Entertainment Europe Ltd Apparatus and method of mapping a virtual environment
GB2571378B (en) * 2018-02-23 2022-04-06 Sony Interactive Entertainment Europe Ltd Apparatus and method of mapping a virtual environment
US11325037B2 (en) 2018-02-23 2022-05-10 Sony Interactive Entertainment Europe Limited Apparatus and method of mapping a virtual environment
US11024054B2 (en) 2019-05-16 2021-06-01 Here Global B.V. Method, apparatus, and system for estimating the quality of camera pose data using ground control points of known quality
CN110162098A (zh) * 2019-07-03 2019-08-23 安徽理工大学 一种矿用无人机
CN110316376A (zh) * 2019-07-03 2019-10-11 安徽理工大学 一种用于探测矿井火灾的无人机
CN113223086A (zh) * 2021-06-09 2021-08-06 司法鉴定科学研究院 适用低质量监控视频的车辆运行状态的重建方法和系统
CN113223086B (zh) * 2021-06-09 2022-05-03 司法鉴定科学研究院 适用低质量监控视频的车辆运行状态的重建方法和系统

Similar Documents

Publication Publication Date Title
CN107292949B (zh) 场景的三维重建方法、装置及终端设备
CN109727288B (zh) 用于单目同时定位与地图构建的系统和方法
WO2018214086A1 (zh) 场景的三维重建方法、装置及终端设备
CN106885574B (zh) 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法
CN110782494A (zh) 一种基于点线融合的视觉slam方法
Luo et al. Real-time dense monocular SLAM with online adapted depth prediction network
KR102367361B1 (ko) 위치 측정 및 동시 지도화 방법 및 장치
Kwon et al. A geometric particle filter for template-based visual tracking
JP2018523881A (ja) データを位置合わせする方法及びシステム
CN113298934B (zh) 一种基于双向匹配的单目视觉图像三维重建方法及系统
CN111127524A (zh) 一种轨迹跟踪与三维重建方法、系统及装置
KR20130084849A (ko) 카메라 추적을 위한 방법 및 장치
KR20160123871A (ko) 영상 광류를 추정하는 방법 및 이를 위한 장치
Ci et al. Stereo visual odometry based on motion decoupling and special feature screening for navigation of autonomous vehicles
Mehralian et al. EKFPnP: extended Kalman filter for camera pose estimation in a sequence of images
Bu et al. Semi-direct tracking and mapping with RGB-D camera for MAV
Cao et al. Parallel K nearest neighbor matching for 3D reconstruction
JP5973767B2 (ja) 対応点探索装置、そのプログラム及びカメラパラメータ推定装置
Li et al. A SLAM system based on RGBD image and point-line feature
WO2024051591A1 (zh) 用于估算视频旋转的方法、装置、电子设备和存储介质
Yang et al. Vision-inertial hybrid tracking for robust and efficient augmented reality on smartphones
Kuschk et al. Real-time variational stereo reconstruction with applications to large-scale dense SLAM
CN113763468B (zh) 一种定位方法、装置、系统及存储介质
Plakas et al. Uncalibrated vision for 3-D underwater applications
Yang et al. Fast and accurate visual odometry from a monocular camera

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17911277

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17911277

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 28.08.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 17911277

Country of ref document: EP

Kind code of ref document: A1