WO2021051220A1 - 一种点云融合方法、设备、系统及存储介质 - Google Patents
一种点云融合方法、设备、系统及存储介质 Download PDFInfo
- Publication number
- WO2021051220A1 WO2021051220A1 PCT/CN2019/105885 CN2019105885W WO2021051220A1 WO 2021051220 A1 WO2021051220 A1 WO 2021051220A1 CN 2019105885 W CN2019105885 W CN 2019105885W WO 2021051220 A1 WO2021051220 A1 WO 2021051220A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image frame
- point cloud
- dimensional point
- adjacent
- cloud corresponding
- Prior art date
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 17
- 230000004927 fusion Effects 0.000 claims abstract description 100
- 238000000034 method Methods 0.000 claims abstract description 46
- 238000007499 fusion processing Methods 0.000 claims abstract description 27
- 238000012545 processing Methods 0.000 claims description 44
- 230000015654 memory Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 description 12
- 238000010586 diagram Methods 0.000 description 9
- 238000013507 mapping Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000001931 thermography Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000000280 densification Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present invention relates to the field of control technology, in particular to a point cloud fusion method, equipment, system and storage medium.
- Point cloud fusion is a key step in 3D reconstruction.
- the existing point cloud fusion scheme needs to reproject all point clouds onto the depth map every time.
- the fusion process will take a long time. . Therefore, how to perform point cloud fusion more quickly and effectively is an important issue that needs to be solved urgently.
- the embodiments of the present invention provide a point cloud fusion method, device, system, and storage medium, which can improve the efficiency and effectiveness of point cloud fusion and reduce resource occupancy.
- an embodiment of the present invention provides a point cloud fusion method, including:
- the three-dimensional point cloud corresponding to the first adjacent image frame is projected onto the depth map of the current image frame to convert the three-dimensional point cloud corresponding to the first adjacent image frame to the three-dimensional point cloud corresponding to the current image frame.
- the point cloud is fused.
- an embodiment of the present invention provides a point cloud fusion device, including a memory and a processor;
- the memory is used to store programs
- the processor is used to call the program, and when the program is executed, it is used to perform the following operations:
- the three-dimensional point cloud corresponding to the first adjacent image frame is projected onto the depth map of the current image frame to convert the three-dimensional point cloud corresponding to the first adjacent image frame to the three-dimensional point cloud corresponding to the current image frame.
- the point cloud is fused.
- an embodiment of the present invention provides a point cloud fusion system, including:
- a movable platform includes a photographing device for photographing the environment to obtain an image frame;
- an embodiment of the present invention provides a computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the method as described in the first aspect is implemented.
- the point cloud fusion device obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position by obtaining the depth map of the current image frame, and combines the first adjacent image frame
- the three-dimensional point cloud corresponding to the image frame is projected onto the depth map of the current image frame to perform fusion processing on the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame.
- FIG. 1 is a schematic diagram of a drone surveying and mapping scene provided by an embodiment of the present invention
- FIG. 2 is a schematic flowchart of a point cloud fusion method provided by an embodiment of the present invention
- FIG. 3 is a schematic diagram of an image frame with adjacent positions provided by an embodiment of the present invention.
- FIG. 4 is a schematic diagram of a point cloud fusion algorithm provided by an embodiment of the present invention.
- FIG. 5 is a schematic flowchart of another point cloud fusion method provided by an embodiment of the present invention.
- Figure 6 is a schematic diagram of a denoising process provided by an embodiment of the present invention.
- FIG. 7 is a schematic flowchart of yet another point cloud fusion method provided by an embodiment of the present invention.
- FIG. 8 is a schematic structural diagram of a point cloud fusion device provided by an embodiment of the present invention.
- FIG. 1 is a schematic diagram of a drone surveying and mapping scene in an embodiment of the present invention.
- the drone surveying and mapping system includes a drone 101 and a ground station 102.
- the drone 101 may specifically be a drone for performing surveying and mapping tasks.
- the UAV 101 may be a multi-rotor UAV.
- it may be a four-rotor UAV, a hexarotor UAV, or an eight-rotor UAV; the UAV 101 may also be a vertical take-off and landing unmanned aerial vehicle.
- a man-machine, the vertical take-off and landing drone has a rotor power system and a fixed-wing power system; the drone 101 may also be a fixed-wing drone.
- the ground station 102 may be a remote control, a smart phone, a tablet computer, a ground control station, a laptop, a watch, a bracelet, etc., and combinations thereof. In this embodiment, the ground station 102 may be specifically as shown in FIG. 1 PC ground station.
- the ground station 102 can determine the flight route information according to the relevant information of the target area.
- the target area can be the area selected by the surveying staff on the user interface of the ground station 102, or the target area can be based on the surveying staff’s input on the ground station 102.
- the information determines the area.
- the ground station 102 sends the flight route information to the drone 101.
- the drone 101 is equipped with a camera through a pan/tilt, and the drone 101 uses the camera to take image frames during the movement according to the flight route information, and sends the image frames to the ground station 102 in real time.
- the ground station 102 may process the image frames to generate a three-dimensional point cloud corresponding to the target area, and display the above-mentioned three-dimensional point cloud in real time while the drone is flying according to the flight route.
- the ground station 102 When the ground station 102 processes the image frame to obtain the 3D point cloud, it needs to merge the 3D point cloud corresponding to the same feature.
- the existing point cloud fusion scheme needs to reproject all the point clouds to the depth each time
- the time to project to the depth map will continue to increase, and the fusion process will become slower and slower, resulting in a freeze when displaying the point cloud in real time and reducing user experience.
- the present invention proposes a point cloud fusion method using three-dimensional point clouds corresponding to adjacent image frames.
- This implementation method projects the three-dimensional point clouds corresponding to the adjacent image frames to the current image frame.
- the number of three-dimensional point clouds for each projection can be basically kept stable, which can overcome the above-mentioned slow fusion speed and low efficiency problems.
- there is a high overlap area between adjacent image frames that is, there are more pixel pairs corresponding to the same feature in adjacent image frames, so point cloud fusion with adjacent image frames can maintain The effectiveness of point cloud fusion.
- the point cloud fusion method provided in the embodiment of the present invention may be executed by a point cloud fusion system, and specifically, may be executed by a point cloud fusion device in the point cloud fusion system.
- the point cloud fusion system includes a point cloud fusion device and a movable platform.
- the point cloud fusion device may be installed on a movable platform; in some embodiments, the point cloud fusion device may be spatially independent from the movable platform; in some embodiments
- the point cloud fusion device may be a component of a movable platform, that is, the movable platform includes a point cloud fusion device.
- the movable platform may be an unmanned robot (such as an unmanned aerial vehicle, an unmanned vehicle, etc.) or a handheld device (such as a handheld pan/tilt camera).
- the point cloud fusion device may be set on a terminal device (for example, a smart phone, a tablet computer, a laptop computer, etc.).
- the movable platform includes a photographing device, and the photographing device is used to photograph the environment to obtain image frames.
- the photographing device may include, but is not limited to, devices such as a visible light camera or a thermal imaging camera.
- FIG. 2 the point cloud fusion method provided by the embodiment of the present invention will be schematically described with reference to FIG. 2 to FIG. 8.
- FIG. 8 the point cloud fusion method provided by the embodiment of the present invention will be schematically described with reference to FIG. 2 to FIG. 8.
- FIG. 2 is a schematic flowchart of a point cloud fusion method provided by an embodiment of the present invention.
- the method may be executed by a point cloud fusion device, where the specific explanation of the point cloud fusion device is as described above.
- the method of the embodiment of the present invention includes the following steps.
- the point cloud fusion device can obtain the depth map of the current image frame.
- the depth map of the current image frame may be calculated by traditional densification algorithms such as semi-global matching (SGM), PatchMatch, PMVS, etc., and machine learning methods MVSNet, DPSnet, etc.
- SGM semi-global matching
- PatchMatch PatchMatch
- PMVS PMVS
- MVSNet machine learning methods
- the point cloud fusion device may also obtain a three-dimensional point cloud corresponding to the current image frame.
- the three-dimensional point cloud corresponding to the current image frame is acquired to facilitate subsequent fusion processing of the three-dimensional point cloud corresponding to the current image frame and the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position.
- the three-dimensional point cloud corresponding to the current image frame may be obtained by a photographing device on a movable platform.
- the photographing device may include, but is not limited to, a visible light camera, a thermal imaging camera, and other devices.
- the point cloud fusion device when the point cloud fusion device acquires the three-dimensional point cloud corresponding to the current image frame, it can convert the pixels in the current image frame into the world coordinate system based on the conversion matrix to obtain the corresponding current image frame. 3D point cloud.
- the conversion matrix includes an internal parameter matrix and an external parameter matrix, and the external parameter matrix includes a rotation matrix and/or a translation vector.
- the external parameter matrix when the origin of the world coordinate system is set on the movable platform, the external parameter matrix only includes a rotation matrix.
- the internal parameter matrix is determined based on a plurality of internal parameters, and the internal parameters are parameters calibrated by the camera, such as focal length, image principal point coordinates, and so on.
- the external parameter matrix may include a rotation matrix and/or a translation vector, wherein the rotation matrix may be determined by the posture of the camera, and the translation vector may be determined by the positioning information of the camera. .
- the embodiment of the present invention may perform the distortion processing on the current image frame before acquiring the three-dimensional point cloud corresponding to the current image frame, thereby improving the accuracy of the position of the three-dimensional point cloud.
- S202 Acquire a three-dimensional point cloud corresponding to a first adjacent image frame adjacent to the current image frame position.
- the point cloud fusion device may obtain the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position.
- the first adjacent image frame adjacent to the current image frame position can be specifically illustrated in FIG. 3 as an example.
- FIG. 3 is a position adjacent image frame provided by an embodiment of the present invention. As shown in FIG. 3, the dotted line is the flight path, and the first adjacent image frame adjacent to the current image frame 30 position includes the first adjacent image frame 31 and the first adjacent image frame 32.
- the image frames adjacent to the position shown in Fig. 3 can be obtained by shooting with the camera mounted on the gimbal.
- the point cloud fusion device when the point cloud fusion device acquires the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position, it may combine each pixel in the first adjacent image frame The points are projected into the three-dimensional space to obtain the three-dimensional point cloud corresponding to the first adjacent image frame.
- the point cloud fusion device when it obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position, it may obtain the three-dimensional point cloud corresponding to the first adjacent image frame And obtain the three-dimensional point cloud corresponding to the first adjacent image frame according to the point cloud mark.
- the point cloud mark may be used to mark the point cloud corresponding to which image frames of the three-dimensional point cloud are fused; in other embodiments, the point cloud mark may also indicate other meanings.
- the marking method of the point cloud marking may include, but is not limited to, numbers, letters, and the like. In this way of point cloud marking, you can intuitively determine which point cloud the 3D point cloud is based on based on the point cloud marking, so as to improve the efficiency of determining the 3D point cloud.
- the adjacent image frames before the 12th frame image include the 10th frame image and the 11th frame image
- the step of projecting the 3D point cloud corresponding to the 11th frame of image is not required at this time. Instead, the 3D point cloud is directly obtained from the previously generated point cloud collection, where the point cloud collection is
- Each of the 3D point clouds will have a corresponding point cloud mark.
- one of the point cloud markers can be (10, 11) to mark that the point cloud is a fusion of the 10th frame image and the 11th frame image.
- one of the point cloud markers can be 11, which is used to mark this
- the point cloud is a newly generated point cloud from the part of the 11th frame image that does not overlap with other images.
- the point cloud corresponding to the 11th frame image needs to be obtained, only the point corresponding to the point cloud mark with 11 needs to be obtained The cloud is fine.
- S203 Project the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame to correspond the three-dimensional point cloud corresponding to the first adjacent image frame to the current image frame
- the 3D point cloud is fused.
- the point cloud fusion device may project the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame, so as to map the three-dimensional point cloud corresponding to the first adjacent image frame to the depth map of the current image frame.
- the point cloud is fused with the three-dimensional point cloud corresponding to the current image frame.
- Figure 4 is a schematic diagram of a point cloud fusion algorithm provided by an embodiment of the present invention.
- the depth map of the current image frame is the depth map 42
- the current image frame The depth map of the adjacent first adjacent image frame is the depth map 41
- the point cloud fusion device may project the three-dimensional point cloud corresponding to the depth map 41 onto the depth map 42 to map the depth map
- the three-dimensional point cloud corresponding to 41 and the three-dimensional point cloud corresponding to the depth map 42 are fused.
- the point cloud fusion device projects the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame
- the position information and posture corresponding to the current image frame can be obtained And project the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame according to the position information and posture information corresponding to the current image frame.
- the depth map of the current image frame is the depth map 42
- the depth map of the first adjacent image frame adjacent to the current image frame position is the depth map 41.
- Pixels can be transferred to the three-dimensional coordinate system; in the depth map 41, the pixel u can get a three-dimensional point cloud p through back projection.
- the point cloud fusion device can project the three-dimensional point cloud set ⁇ P ⁇ onto the depth map 42 through T.
- the back projection is a process of projecting two-dimensional pixels into a three-dimensional space to obtain a three-dimensional point cloud.
- the point cloud fusion device before the point cloud fusion device performs fusion processing on the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame, it may acquire the first adjacent image frame.
- the projection area obtained by projecting the three-dimensional point cloud corresponding to the image frame onto the depth map of the current image frame is determined according to the depth information of the three-dimensional point cloud corresponding to the first adjacent image frame and the depth information of the projection area The effective three-dimensional point cloud of the current image frame.
- the first adjacent image frame may be The three-dimensional point cloud corresponding to the image frame is fused with the effective three-dimensional point cloud of the current image frame.
- the depth map of the current image frame is the depth map 42
- the depth map of the first adjacent image frame adjacent to the current image frame position is the depth map 41.
- Pixels can be transferred to the three-dimensional coordinate system; in the depth map 41, the pixel u can get a three-dimensional point cloud p through back projection.
- the point cloud set P ⁇ p ⁇ .
- the three-dimensional point cloud set ⁇ P ⁇ can be projected onto the depth map 42 through T.
- the pixel u in the depth map 41 corresponds to the three-dimensional point cloud p, and p'can be obtained through the pose transformation T, and p'can be projected into the depth map 42 to obtain u', which can be further determined
- the projection area around u'can be a 5x5 pixel projection area.
- the point cloud fusion device can determine the effective three-dimensional point of the depth map 42 according to the depth information of u in the depth map 41 and the depth information of the 5x5 projection area.
- a first preset threshold for example 2%
- the three-dimensional point cloud corresponding to pixel point a can be mapped to pixel point u
- the 3D point cloud is processed for point cloud fusion.
- the point cloud fusion device may acquire the first adjacent image frame when acquiring the projection area obtained by projecting the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame.
- the three-dimensional point cloud corresponding to the adjacent image frame is projected onto the projection point obtained on the depth map of the current image frame, and the projection area is determined with the projection point as the center.
- the depth map of the current image frame is the depth map 42
- the depth map of the first adjacent image frame adjacent to the current image frame position is the depth map 41.
- the pixel can be transferred to the three-dimensional coordinate system; in the depth map 41, the pixel u can get a three-dimensional point cloud p through back projection, when all the pixels in the depth map 41 are converted into a three-dimensional point cloud ,
- the three-dimensional point cloud set ⁇ P ⁇ can be projected onto the depth map 42 through T to obtain u', and the 5x5 area with u'as the center is determined to be The projection area.
- the point cloud fusion device determines the effective three-dimensional point cloud of the current image frame according to the depth information of the three-dimensional point cloud corresponding to the first adjacent image frame and the depth information of the projection area , Can obtain the depth information of the three-dimensional point cloud corresponding to the first adjacent image frame and the depth information of each pixel in the projection area, and calculate the depth information of the three-dimensional point cloud corresponding to the first adjacent image frame And the depth difference of the depth information of each pixel in the projection area to determine that the three-dimensional point cloud corresponding to the pixel whose depth difference is less than the first preset threshold is an effective three-dimensional point cloud.
- the depth map of the current image frame is the depth map 42
- the depth map of the first adjacent image frame adjacent to the current image frame position is the depth map 41.
- Pixels can be transferred to the three-dimensional coordinate system; in the depth map 41, the pixel u can get a three-dimensional point cloud p through back projection.
- you can Obtain the point cloud set P ⁇ p ⁇ .
- the three-dimensional point cloud set ⁇ P ⁇ can be projected onto the depth map 42 through T to obtain u', and a 5x5 projection area with u'as the center is determined
- the three-dimensional point cloud corresponding to the pixel point a is an effective three-dimensional point cloud.
- the effective three-dimensional point cloud is determined through this implementation, and the three-dimensional point cloud corresponding to the first adjacent image frame is fused with the effective three-dimensional point cloud of the current image frame, which can improve the effectiveness of point cloud fusion. Sex.
- the point cloud fusion device when it performs fusion processing on the three-dimensional point cloud corresponding to the first adjacent image frame and the effective three-dimensional point cloud of the current image frame, it may perform fusion processing according to the first phase.
- the three-dimensional point cloud corresponding to the adjacent image frame and the effective three-dimensional point cloud of the current image frame generate a fusion point cloud; wherein the depth information of the fusion point cloud is based on the three-dimensional point cloud corresponding to the first adjacent image frame
- the depth information is obtained from the depth information of the effective three-dimensional point cloud of the current image frame.
- the fusion point cloud can be generated directly based on the depth information of the 3D point cloud corresponding to the 11th frame image and the depth information of the effective 3D point cloud in the 12th frame image.
- the point cloud fusion device obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position by obtaining the depth map of the current image frame, and combines the first adjacent image frame
- the three-dimensional point cloud corresponding to the image frame is projected onto the depth map of the current image frame to perform fusion processing on the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame.
- FIG. 5 is a schematic flowchart of another point cloud fusion method provided by an embodiment of the present invention.
- the method can be executed by a point cloud fusion device.
- the specific explanation of the point cloud fusion device is as described above. .
- the difference between the method described in the embodiment of the present invention and the method described in FIG. 2 is that the embodiment of the present invention describes in detail the process of performing denoising processing on the depth map of the image frame. Specifically, the method described in the embodiment of the present invention Including the following steps.
- S501 Acquire a depth map of the current image frame.
- the point cloud fusion device can obtain the depth map of the current image frame.
- S502 Acquire a three-dimensional point cloud corresponding to a target image frame that meets a preset condition.
- the point cloud fusion device can obtain a three-dimensional point cloud corresponding to a target image frame that meets a preset condition.
- the meeting the preset condition includes that the target image frame is any previous image frame adjacent to the current image frame position.
- the time when the target image frame is taken is before the time when the current image frame is taken, and the position of the target image frame is adjacent to the current image frame.
- the position adjacent means that the position of the drone when the target image frame is taken is adjacent to the position of the drone when the current image frame is taken. For example, the position of the drone when the target image frame is taken is Point A, the position of the drone when shooting the current image frame is point B, then the distance between point A and point B is within the preset distance interval.
- the image frame before the current image frame has been acquired. There may be a new scene in the current image frame that is different from the previously acquired image frame.
- the point cloud is subjected to denoising processing, and the pixels corresponding to these new scenes may be filtered out regardless of whether the depth value is reasonable or not, causing some reasonable 3D point clouds to be filtered out.
- the embodiment of the present invention applies a certain lag to the image frame undergoing fusion processing, that is, assuming that the current depth map has been calculated to the kth frame, that is, the current image frame is the kth frame, then the kth frame is Before the frame is fused, the 3D point cloud of the kn frame adjacent to the k-th frame position needs to be denoised.
- the n is any value smaller than k; in an example, n may be 1.
- the target image frame may be the same as the 12th frame image.
- Frame image position adjacent to the 11th frame image; or, the target image frame may be the 10th frame image adjacent to the 12th frame image position; or, the target image frame may be the 12th frame image position The adjacent 9th frame image.
- the point cloud fusion device when the point cloud fusion device obtains a three-dimensional point cloud corresponding to a target image frame that meets a preset condition according to the depth map of the current image frame, it may project each pixel in the target image frame to In a three-dimensional space, a three-dimensional point cloud corresponding to the target image frame is obtained.
- S503 Perform denoising processing on the three-dimensional point cloud corresponding to the target image frame.
- the point cloud fusion device may perform denoising processing on the three-dimensional point cloud corresponding to the target image frame.
- the target image frame when performing denoising processing in the present invention, may be determined as the target image frame according to the position information of the image frame, such as Global Positioning System (GPS) information or the position information obtained through aerial triangulation.
- GPS Global Positioning System
- the three-dimensional point cloud corresponding to the target image frame is projected onto the depth map of the image frame adjacent to the target image frame to perform denoising processing according to depth consistency.
- the entire projection process is completely parallel, and the depth map of the image frame can be subdivided into multiple sub-regions concurrently, and the parallel processor is used for acceleration to ensure its real-time performance.
- the point cloud fusion device when it performs denoising processing on the three-dimensional point cloud corresponding to the target image frame, it can obtain the depth map and the first image frame of the third adjacent image frame adjacent to the position of the target image frame.
- Four depth maps of adjacent image frames and performing denoising processing on the three-dimensional point cloud corresponding to the target image frame according to the depth map of the third adjacent image frame and the depth map of the fourth adjacent image frame.
- the point cloud fusion device when the point cloud fusion device performs denoising processing on the three-dimensional point cloud corresponding to the target image frame according to the depth map of the third adjacent image frame and the depth map of the fourth adjacent image frame,
- the three-dimensional point cloud corresponding to the target image frame may be projected onto the depth map of the third adjacent image frame to obtain a first projection area, and the three-dimensional point cloud corresponding to the target image frame and the first projection may be obtained The first depth difference of the region; and project the three-dimensional point cloud corresponding to the target image frame onto the depth map of the fourth adjacent image frame to obtain a second projection region, and obtain the three-dimensional point cloud corresponding to the target image frame
- the second depth difference between the point cloud and the second projection area thereby performing denoising processing on the three-dimensional point cloud corresponding to the target image frame according to the first depth difference and the second depth difference.
- the target image when the point cloud fusion device performs denoising processing on the three-dimensional point cloud corresponding to the target image frame according to the first depth difference and the second depth difference, the target image can be obtained
- FIG. 6 is a schematic diagram of a denoising processing provided by an embodiment of the present invention.
- the depth map 62 corresponds to the current image frame
- the depth map 61 is the target image frame
- the depth map of the third adjacent image frame adjacent to the target image frame 61 is the depth map 60 and the depth map 62.
- the pixel point u in the depth map 61 corresponding to the target image frame corresponds to the three-dimensional point cloud p in the three-dimensional space.
- Projecting the three-dimensional point cloud p onto the depth map 60 can obtain a projection area, which can be specifically As the first projection area centered on the projection point u, a projection area can be obtained by projecting the three-dimensional point cloud p onto the depth map 62, which may specifically be a second projection area centered on the projection point u.
- the 3D point cloud p corresponding to the pixel u in the depth map 61 can be retained, and vice versa, the 3D point cloud p corresponding to the pixel u in the depth map 61 can be deleted to complete the denoising process .
- the depth value of the pixel point u is considered to be unreliable, that is, the pixel point u corresponds to The position of the 3D point cloud is unreliable.
- this method of denoising the three-dimensional point cloud corresponding to the target image frame can filter invalid three-dimensional point clouds, reduce the overhead of noise in the fusion process, reduce the memory usage, and help improve the point. Effectiveness and efficiency of cloud integration.
- S504 Acquire a three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position after denoising processing.
- the point cloud fusion device may obtain the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position after denoising processing.
- the specific implementation is as described above and will not be repeated here.
- S505 Project the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame to correspond the three-dimensional point cloud corresponding to the first adjacent image frame to the current image frame
- the 3D point cloud is fused.
- the point cloud fusion device may project the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame, so as to map the three-dimensional point cloud corresponding to the first adjacent image frame to the depth map of the current image frame.
- the point cloud is fused with the three-dimensional point cloud corresponding to the current image frame.
- the point cloud fusion device can obtain a three-dimensional point cloud corresponding to a target image frame that meets preset conditions, perform denoising processing on the three-dimensional point cloud corresponding to the target image frame, and obtain the denoising processed and
- the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position is projected onto the depth map of the current image frame to project the three-dimensional point cloud corresponding to the first adjacent image frame.
- the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame are fused.
- FIG. 7 is a schematic flowchart of another point cloud fusion method provided by an embodiment of the present invention.
- the method can be executed by a point cloud fusion device, where the specific explanation of the point cloud fusion device is as described above. .
- the difference between the method described in the embodiment of the present invention and the method described in FIG. 5 is that the embodiment of the present invention describes in detail the fusion processing process of the previous image frame adjacent to the current image frame. Specifically, the embodiment of the present invention The method includes the following steps.
- S701 Acquire a depth map of a previous image frame adjacent to the current image frame.
- the point cloud fusion device may obtain the depth map of the previous image frame adjacent to the current image frame.
- the point cloud fusion device may acquire the depth map of the previous image frame adjacent to the current image frame.
- the point cloud fusion device can obtain the 11th frame image adjacent to the 12th frame image Depth map.
- the point cloud fusion device after acquiring the depth map of the previous image frame adjacent to the current image frame, acquires the image corresponding to the second adjacent image frame adjacent to the position of the previous image frame. Before the three-dimensional point cloud, the three-dimensional point cloud corresponding to the target image frame that meets the preset conditions can be obtained, and the three-dimensional point cloud corresponding to the target image frame can be denoised. In some embodiments, the meeting the preset condition includes that the target image frame is any previous image frame that is adjacent to the previous image frame.
- the point cloud fusion device can determine the 10th frame image adjacent to the 11th frame image position as the target Image frames, and obtain the three-dimensional point cloud corresponding to the tenth frame image, and perform denoising processing on the three-dimensional point cloud corresponding to the tenth frame image.
- S702 Acquire a three-dimensional point cloud corresponding to a second adjacent image frame adjacent to the position of the previous image frame.
- the point cloud fusion device can obtain the three-dimensional point cloud corresponding to the second adjacent image frame adjacent to the position of the previous image frame.
- the point cloud fusion device can obtain the image corresponding to the 10th frame image adjacent to the position of the 11th frame image.
- Three-dimensional point cloud can obtain the image corresponding to the 10th frame image adjacent to the position of the 11th frame image.
- S703 Project the three-dimensional point cloud corresponding to the second adjacent image frame onto the depth map of the previous image frame, so as to compare the three-dimensional point cloud corresponding to the second adjacent image frame with the previous image
- the three-dimensional point cloud corresponding to the frame is fused.
- the point cloud fusion device may project the three-dimensional point cloud corresponding to the second adjacent image frame onto the depth map of the previous image frame, so as to map the corresponding three-dimensional point cloud of the second adjacent image frame to the depth map of the previous image frame.
- the three-dimensional point cloud is fused with the three-dimensional point cloud corresponding to the previous image frame.
- the point cloud fusion device can The three-dimensional point cloud corresponding to the 10th frame image and the three-dimensional point cloud corresponding to the 11th frame image are fused.
- the specific implementation is as described above and will not be repeated here.
- the flow of the point cloud fusion method shown in FIG. 7 is one of the previous cycles of the flow of the point cloud fusion method shown in FIG. 2.
- the UAV sends the captured image frames to the ground station in real time while flying along the route, and the ground station generates a three-dimensional point cloud in real time based on these image frames after receiving the image frames.
- the 3D point cloud corresponding to the part of the new image frame that overlaps the previously acquired image frame needs to be fused with the previously generated 3D point cloud, and the part that does not overlap needs to be generated.
- Point cloud In this way, when the drone is flying along the route, as the drone captures more and more image frames, the ground station generates more and more 3D point clouds in real time, gradually covering the target area to depict the target Three-dimensional information of the area.
- FIG. 8 is a schematic structural diagram of a point cloud fusion device provided by an embodiment of the present invention.
- the point cloud fusion device includes: a memory 801 and a processor 802.
- the point cloud fusion device further includes a data interface 803, and the data interface 803 is used to transfer data information between the point cloud fusion device and other devices.
- the memory 801 may include a volatile memory (volatile memory); the memory 801 may also include a non-volatile memory (non-volatile memory); the memory 801 may also include a combination of the foregoing types of memories.
- the processor 802 may be a central processing unit (CPU).
- the processor 802 may further include a hardware chip.
- the above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof.
- the above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or any combination thereof.
- the memory 801 is used to store programs, and the processor 802 can call the programs stored in the memory 801 to perform the following steps:
- the three-dimensional point cloud corresponding to the first adjacent image frame is projected onto the depth map of the current image frame to convert the three-dimensional point cloud corresponding to the first adjacent image frame to the three-dimensional point cloud corresponding to the current image frame.
- the point cloud is fused.
- the processor 802 obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position, it is specifically configured to:
- the processor 802 obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position, it is specifically configured to:
- a three-dimensional point cloud corresponding to the first adjacent image frame is acquired.
- the processor 802 projects the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame, it is specifically configured to:
- the processor 802 is further configured to:
- the processor 802 When the processor 802 performs fusion processing on the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame, it is specifically configured to:
- Fusion processing is performed on the three-dimensional point cloud corresponding to the first adjacent image frame and the effective three-dimensional point cloud of the current image frame.
- the processor 802 obtains the projection area obtained by projecting the three-dimensional point cloud corresponding to the first adjacent image frame onto the depth map of the current image frame, it is specifically configured to:
- the projection area is determined with the projection point as the center.
- the processor 802 determines the effective three-dimensional point cloud of the current image frame according to the depth information of the three-dimensional point cloud corresponding to the first adjacent image frame and the depth information of the projection area, it is specifically configured to:
- the three-dimensional point cloud corresponding to the pixel point whose depth difference is less than the first preset threshold is a valid three-dimensional point cloud.
- the processor 802 when the processor 802 performs fusion processing on the three-dimensional point cloud corresponding to the first adjacent image frame and the effective three-dimensional point cloud of the current image frame, it is specifically configured to:
- the depth information of the fusion point cloud is obtained based on the depth information of the three-dimensional point cloud corresponding to the first adjacent image frame and the depth information of the effective three-dimensional point cloud of the current image frame.
- the processor 802 obtains the depth map of the current image frame, before obtaining the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position, it is further used for:
- Denoising processing is performed on the three-dimensional point cloud corresponding to the target image frame.
- the meeting the preset condition includes that the target image frame is any previous image frame adjacent to the current image frame position.
- the processor 802 obtains a three-dimensional point cloud corresponding to a target image frame that meets a preset condition, it is specifically configured to:
- processor 802 when the processor 802 performs denoising processing on the three-dimensional point cloud corresponding to the target image frame, it is specifically configured to:
- processor 802 performs denoising processing on the three-dimensional point cloud corresponding to the target image frame according to the depth map of the third adjacent image frame and the depth map of the fourth adjacent image frame, specifically using in:
- denoising processing is performed on the three-dimensional point cloud corresponding to the target image frame.
- the processor 802 when the processor 802 performs denoising processing on the three-dimensional point cloud corresponding to the target image frame according to the first depth difference value and the second depth difference value, it is specifically configured to:
- processor 802 is further configured to:
- the processor 802 obtains the depth map of the previous image frame adjacent to the current image frame, obtains the three-dimensional point cloud corresponding to the second adjacent image frame adjacent to the position of the previous image frame Previously, it was also used for:
- Denoising processing is performed on the three-dimensional point cloud corresponding to the target image frame.
- the satisfaction of the preset condition includes that the target image frame is any previous image frame adjacent to the previous image frame.
- the point cloud fusion device obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position by obtaining the depth map of the current image frame, and combines the first adjacent image frame
- the three-dimensional point cloud corresponding to the image frame is projected onto the depth map of the current image frame to merge the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame.
- An embodiment of the present invention also provides a point cloud fusion system, the system includes: a movable platform, the movable platform includes a photographing device for photographing an environment to obtain image frames; and the above-mentioned point cloud fusion device.
- the point cloud fusion device obtains the three-dimensional point cloud corresponding to the first adjacent image frame adjacent to the current image frame position by obtaining the depth map of the current image frame, and combines the first adjacent image frame
- the three-dimensional point cloud corresponding to the image frame is projected onto the depth map of the current image frame to merge the three-dimensional point cloud corresponding to the first adjacent image frame and the three-dimensional point cloud corresponding to the current image frame.
- the embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the present invention corresponds to FIG. 2, FIG. 5, or FIG. 7
- the method described in the embodiment can also implement the device corresponding to the embodiment of the present invention described in FIG. 8, and details are not described herein again.
- the computer-readable storage medium may be an internal storage unit of the device described in any of the foregoing embodiments, for example, a hard disk or a memory of the device.
- the computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a Smart Media Card (SMC), or a Secure Digital (SD) card. , Flash Card, etc.
- the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.
- the computer-readable storage medium is used to store the computer program and other programs and data required by the terminal.
- the computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Processing Or Creating Images (AREA)
Abstract
一种点云融合方法、设备、系统及存储介质，其中，该方法包括：获取当前图像帧的深度图（S201）；获取与当前图像帧位置相邻的第一相邻图像帧对应的三维点云（S202）；将第一相邻图像帧对应的三维点云投影到当前图像帧的深度图上，以将第一相邻图像帧对应的三维点云与当前图像帧对应的三维点云进行融合处理（S203）。可以提高点云融合的效率和有效性、降低资源占用率。
Description
本发明涉及控制技术领域，尤其涉及一种点云融合方法、设备、系统及存储介质。
点云融合是三维重建中一个关键的步骤，现有的点云融合方案每次都需要把所有的点云重新投影到深度图上，当点云数量比较多时，融合过程需要比较久的耗时。因此，如何更有快速有效地进行点云融合是亟需解决的重要问题。
发明内容
本发明实施例提供了一种点云融合方法、设备、系统及存储介质，可以提高点云融合的效率和有效性、降低资源占用率。
第一方面，本发明实施例提供了一种点云融合方法，包括：
获取当前图像帧的深度图；
获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云；
将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
第二方面，本发明实施例提供了一种点云融合设备，包括存储器和处理器；
所述存储器，用于存储程序；
所述处理器，用于调用所述程序，当所述程序被执行时，用于执行以下操作：
获取当前图像帧的深度图；
获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云；
将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
第三方面，本发明实施例提供了一种点云融合系统，包括：
可移动平台，所述可移动平台包括拍摄装置，用于对环境进行拍摄得到图像帧；
如上述第二方面所述的点云融合设备。
第四方面，本发明实施例提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现如上述第一方面所述的方法。
本发明实施例中，点云融合设备通过获取当前图像帧的深度图，获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，并将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。通过这种实施方式可以提高点云融合的效率和有效性、降低资源占用率。
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例提供的一种无人机测绘场景的示意图；
图2是本发明实施例提供的一种点云融合方法的流程示意图；
图3是本发明实施例提供的一种位置相邻图像帧的示意图；
图4是本发明实施例提供的一种点云融合算法的示意图；
图5是本发明实施例提供的另一种点云融合方法的流程示意图；
图6是本发明实施例提供的一种去噪处理的示意图；
图7是本发明实施例提供的又一种点云融合方法的流程示意图；
图8是本发明实施例提供的一种点云融合设备的结构示意图。
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动 前提下所获得的所有其他实施例，都属于本发明保护的范围。
下面结合附图，对本发明的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
请参照图1，图1为本发明一个实施例中的无人机测绘场景的示意图，无人机测绘系统包括无人机101，地面站102。无人机101具体可以是执行测绘任务的无人机。可选的，无人机101可以是多旋翼无人机，示例的，可以是四旋翼无人机、六旋翼无人机、八旋翼无人机；无人机101还可以是垂直起降无人机，该垂直起降无人机上具有旋翼动力系统和固定翼动力系统；无人机101还可以是固定翼无人机。地面站102可以是遥控器、智能手机、平板电脑、地面控制站、膝上型电脑、手表、手环等及其组合，在本实施例中，地面站102具体可以是如图1所示的PC地面站。地面站102可以根据目标区域的相关信息，确定飞行航线信息，该目标区域可以是测绘人员在地面站102的用户界面上选择的区域，或者该目标区域可以是根据测绘人员在地面站102上输入的信息确定的区域。地面站102将飞行航线信息发送给无人机101。无人机101通过云台搭载有拍摄装置，无人机101根据该飞行航线信息运动的过程中通过拍摄装置拍摄图像帧，并将图像帧实时发送给地面站102。地面站102可对图像帧进行处理从而生成目标区域对应的三维点云，并在无人机按照飞行航线飞行的过程中对上述三维点云进行实时显示。
地面站102对图像帧进行处理以得到三维点云时，需要将对应于同一地物的三维点云进行融合处理，现有的点云融合方案每次都需要把所有的点云重新投影到深度图上，当点云数量越来越多时，投影到深度图的时间就会不断增长，融合过程越来越慢，导致实时显示点云时出现卡顿、降低用户体验。
针对上述问题，本发明提出一种用位置相邻的图像帧对应的三维点云进行点云融合的方式，这种实施方式通过将位置相邻的图像帧对应的三维点云投影至当前图像帧的深度图以实现点云融合，每次投影的三维点云的数量基本能保持稳定，从而可以克服上述融合速度慢，效率低的问题。并且位置相邻的图像帧之间具有较高的重叠区域，也即位置相邻的图像帧存在较多对应于同一地物的像素对，因此用位置相邻的图像帧进行点云融合可以保持点云融合的有效性。
本发明实施例中提供的点云融合方法可以由一种点云融合系统执行，具体 的，可以由点云融合系统中的点云融合设备执行。其中，所述点云融合系统包括点云融合设备和可移动平台。在某些实施例中，所述点云融合设备可以安装在可移动平台上；在某些实施例中，所述点云融合设备可以在空间上独立于可移动平台；在某些实施例中，所述点云融合设备可以是可移动平台的部件，即所述可移动平台包括点云融合设备。在某些实施例中，所述可移动平台可以是无人控制机器人(例如无人飞行器、无人车等)或者手持式设备(例如手持云台相机)。在某些实施例中，所述点云融合设备可以设置在终端设备(例如智能手机、平板电脑、膝上型电脑等)上。在某些实施例中，所述可移动平台包括拍摄装置，所述拍摄装置用于对环境进行拍摄以得到图像帧。在某些实施例中，所述拍摄装置可以包括但不限于可见光相机或热成像相机等装置。
下面结合附图2-附图8对本发明实施例提供的点云融合方法进行示意性说明。
具体请参见图2，图2是本发明实施例提供的一种点云融合方法的流程示意图，所述方法可以由点云融合设备执行，其中，点云融合设备的具体解释如前所述。具体地，本发明实施例的所述方法包括如下步骤。
S201：获取当前图像帧的深度图。
本发明实施例中，点云融合设备可以获取当前图像帧的深度图。
在某些实施例中，所述当前图像帧的深度图可以通过传统稠密化算法如半全局匹配算法(Semi Global Matching，SGM)，PatchMatch，PMVS等以及机器学习类方法MVSNet，DPSnet等计算得到。
在一个实施例中，所述点云融合设备还可以获取所述当前图像帧对应的三维点云。本发明实施例通过获取当前图像帧对应的三维点云，以便于后续将当前图像帧对应的三维点云与当前图像帧位置相邻的第一相邻图像帧对应的三维点云进行融合处理。
在一些实施例中，所述当前图像帧对应的三维点云可以通过可移动平台上的拍摄装置得到。在某些实施例中，所述拍摄装置可以包括但不限于可见光相机、热成像相机等装置。
在一些实施例中，所述点云融合设备在获取当前图像帧对应的三维点云时，可以基于转换矩阵将当前图像帧中的像素点转换到世界坐标系中，得到所述当前图像帧对应的三维点云。在某些实施例中，所述转换矩阵包括内参矩阵 和外参矩阵，所述外参矩阵包括旋转矩阵和/或平移向量。在某些实施例中，当所述世界坐标系的原点设定在所述可移动平台上时，所述外参矩阵只包括旋转矩阵。
在某些实施例中，所述内参矩阵是根据多个内参数确定得到，所述内参数是相机摄像头标定得到的参数，如焦距、像主点坐标等。在某些实施例中，所述外参矩阵可以包括旋转矩阵和/或平移向量，其中，所述旋转矩阵可以通过相机摄像头的姿态确定得到的，所述平移向量可以通过摄像头的定位信息确定得到。
本发明实施例在获取当前图像帧对应的三维点云之前可以对当前图像帧进行去畸变处理，从而提升三维点云位置的准确性。
S202：获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云。
本发明实施例中，点云融合设备可以获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云。
在一个实施例中，所述与所述当前图像帧位置相邻的第一相邻图像帧，具体可以图3为例进行说明，图3是本发明实施例提供的一种位置相邻图像帧的示意图，如图3所示，虚线为飞行航线，与当前图像帧30位置相邻的第一相邻图像帧包括第一相邻图像帧31和第一相邻图像帧32。无人机在沿飞行航线飞行过程中，可以通过挂载在云台上的拍摄装置拍摄得到如图3所示位置相邻的图像帧。
在一个实施例中，点云融合设备在获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云时，可以将所述第一相邻图像帧中的每个像素点投影到三维空间，得到所述第一相邻图像帧对应的三维点云。
在一个实施例中，点云融合设备在获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云时，可以获取所述第一相邻图像帧对应的三维点云的点云标记，并根据所述点云标记，获取所述第一相邻图像帧对应的三维点云。在某些实施例中，所述点云标记可以用于标记三维点云是由哪些图像帧对应的点云融合得到；在其他实施例中，所述点云标记还可以表示其他含义，本发明实施例不做具体限定。在某些实施例中，所述点云标记的标记方式可以包括但不限于数字、字母等。通过这种点云标记的方式，可以直观地根据点云标记确 定三维点云是根据哪些点云融合得到的，以提高确定三维点云的效率。
例如，假设获取到的当前图像帧为第12帧图像，在第12帧图像之前的位置相邻的图像帧包括第10帧图像和第11帧图像，如果第11帧图像对应的三维点云是已经经过了融合的，此时获取第11帧图像对应的三维点云则不需要投影至三维空间的步骤，而是直接在之前生成的点云集合中获取三维点云，其中，点云集合中的每个三维点云都会有对应的点云标记。例如，其中一个点云标记可以是(10、11)用于标记这个点云是由第10帧图像和第11帧图像融合的，又例如，其中一个点云标记可以是11，用于标记这个点云是由第11帧图像中不与其他图像重叠的部分新产生的点云，如此，当需要获取第11帧图像对应的点云时，只需要获取带有11的点云标记对应的点云即可。
S203：将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
本发明实施例中，点云融合设备可以将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
具体可以图4为例进行说明，图4是本发明实施例提供的一种点云融合算法的示意图；如图4所示，其中，当前图像帧的深度图为深度图42，与当前图像帧位置相邻的第一相邻图像帧的深度图为深度图41，则点云融合设备可以将所述深度图41对应的三维点云投影到所述深度图42上，以将所述深度图41对应的三维点云与所述深度图42对应的三维点云进行融合处理。
在一个实施例中，点云融合设备将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上时，可以获取所述当前图像帧对应的位置信息和姿态信息，并根据所述当前图像帧对应的位置信息和姿态信息，将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上。
如图4所示，当前图像帧的深度图为所述深度图42，与当前图像帧位置相邻的第一相邻图像帧的深度图为深度图41，对于每个深度图中的每个像素，可以把像素点转到三维坐标系中；在深度图41中，像素点u可以通过反向投影得到一个三维点云p，当深度图41中所有像素点都转化为三维点云后，可以得到点云集P＝{p}。假设深度图41和深度图42之间的位姿变换为T，则 点云融合设备可以通过T将三维点云集{P}投影到深度图42上。在某些实施例中，所述反向投影是将二维像素投影到三维空间得到三维点云的过程。
在一个实施例中，所述点云融合设备将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理之前，可以获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域，并根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云。在一种实施方式中，点云融合设备在将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理时，可以将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理。
如图4所示，当前图像帧的深度图为所述深度图42，与当前图像帧位置相邻的第一相邻图像帧的深度图为深度图41，对于每个深度图中的每个像素，可以把像素转到三维坐标系中；在深度图41中，像素点u可以通过反向投影得到一个三维点云p，当深度图41中所有像素点都转化为三维点云后，可以得到点云集P＝{p}。假设深度图41和深度图42之间的位姿变换为T，则通过T可以将三维点云集{P}投影到深度图42上。以一个像素点为例，深度图41中的像素点u对应于三维点云p，通过位姿变换T可得到p'，将p'投影至深度图42中得到u'，进一步地可以确定出u'周围的投影区域，具体可以为5x5像素的投影区域，点云融合设备可以根据深度图41中u的深度信息和所述5x5的投影区域的深度信息确定所述深度图42的有效三维点云，如果在这个投影区域中的某像素点a的深度值和u的深度值之差小于第一预设阈值，例如2％，则可以将像素点a对应的三维点云和像素点u对应的三维点云进行点云融合处理。
在一个实施例中，所述点云融合设备在获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域时，可以获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影点，并以所述投影点为中心确定所述投影区域。
例如，如图4所示，当前图像帧的深度图为所述深度图42，与当前图像帧位置相邻的第一相邻图像帧的深度图为深度图41，对于每个深度图中的每个像素，可以把像素转到三维坐标系中；在深度图41中，像素点u可以通过 反向投影得到一个三维点云p，当深度图41中所有像素点都转化为三维点云后，可以得到点云集P＝{p}。假设深度图41和深度图42之间的位姿变换为T，则通过T可以将三维点云集{P}投影到深度图42上得到u'，并确定出u'为中心的5x5的区域为所述投影区域。
在一个实施例中，所述点云融合设备在根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云时，可以获取所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域中各个像素点的深度信息，并计算所述第一相邻图像帧对应的三维点云的深度信息与所述投影区域中各个像素点的深度信息的深度差值，以确定所述深度差值小于第一预设阈值的所述像素点对应的三维点云为有效三维点云。
如图4所示，当前图像帧的深度图为所述深度图42，与当前图像帧位置相邻的第一相邻图像帧的深度图为深度图41，对于每个深度图中的每个像素，可以把像素转到三维坐标系中；在深度图41中，像素点u可以通过反向投影得到一个三维点云p，当深度图41中所有像素点都转化为三维点云后，可以得到点云集P＝{p}。假设深度图41和深度图42之间的位姿变换为T，则通过T可以将三维点云集{P}投影到深度图42上得到u'，并确定出u'为中心的5x5的投影区域，当这个投影区域中的某像素点a的深度值和p'的深度差小于第一预设阈值2％时，像素点a对应的三维点云为有效三维点云。
可见，通过这种实施方式确定出有效三维点云，将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理，可以提高点云融合的有效性。
在一个实施例中，所述点云融合设备在将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理时，可以根据所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云生成融合点云；其中，所述融合点云的深度信息是根据所述第一相邻图像帧对应的三维点云的深度信息与所述当前图像帧的有效三维点云的深度信息得到的。
例如，假设当前图像帧为第12帧图像，与所述当前图像帧第12帧图像位置相邻的第一相邻图像帧为第11帧图像，如果已经确定出第12帧图像中的有效三维点云，则可以直接根据第11帧图像对应的三维点云的深度信息与第12帧图像中的有效三维点云的深度信息生成融合点云。
可见，通过这种先确定出当前图像帧的有效三维点云，然后直接将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理的实施方式，进一步提高了点云融合的有效性和效率。
本发明实施例中，点云融合设备通过获取当前图像帧的深度图，获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，并将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。通过这种实施方式可以提高点云融合的效率和有效性、降低资源占用率。
具体请参见图5，图5是本发明实施例提供的另一种点云融合方法的流程示意图，所述方法可以由点云融合设备执行，其中，点云融合设备的具体解释如前所述。本发明实施例所述的方法与图2所述方法的区别在于，本发明实施例是对图像帧的深度图进行去噪处理的过程进行详细说明，具体地，本发明实施例的所述方法包括如下步骤。
S501：获取当前图像帧的深度图。
本发明实施例中，点云融合设备可以获取当前图像帧的深度图。
S502：获取满足预设条件的目标图像帧对应的三维点云。
本发明实施例中，点云融合设备可以获取满足预设条件的目标图像帧对应的三维点云。在某些实施例中，所述满足预设条件包括所述目标图像帧是所述当前图像帧位置相邻的前任一图像帧。在某些实施例中，拍摄所述目标图像帧的时间位于拍摄所述当前图像帧的时间之前，且目标图像帧与当前图像帧位置相邻。在某些实施例中，所述位置相邻是指拍摄目标图像帧时无人机的位置与拍摄当前图像帧时无人机的位置相邻，例如拍摄目标图像帧时无人机的位置为A点，拍摄当前图像帧时无人机的位置为B点，那么A点和B点之间的距离在预设距离间隔范围之内。
在一个实施例中，在获取到当前图像帧时，已经获取到当前图像帧之前的图像帧，当前图像帧中可能存在区别于之前获取的图像帧的新的场景，如果对当前图像帧的三维点云进行去噪处理，这些新的场景对应的像素点无论深度值是否合理都可能会被滤掉，造成一些合理的三维点云被过滤掉。针对这种情况，本发明实施例把进行融合处理的图像帧进行一定的滞后，即假设现在的深度图 已经计算到第k帧了，也即当前图像帧为第k帧，那么在对第k帧进行融合处理之前，需对与第k帧位置相邻的k-n帧的三维点云进行去噪处理。在某些实施例中，所述n为小于k的任意数值；在一个示例中，n可以为1。
例如，假设当前图像帧为第12帧图像，与第12帧图像位置相邻的图像帧包括第11帧图像、第10帧图像和第9帧图像，则所述目标图像帧可以是与第12帧图像位置相邻的第11帧图像；或者，所述目标图像帧可以是与第12帧图像位置相邻的第10帧图像；又或者，所述目标图像帧可以是与第12帧图像位置相邻的第9帧图像。
在一个实施例中，点云融合设备根据所述当前图像帧的深度图获取满足预设条件的目标图像帧对应的三维点云时，可以将所述目标图像帧中的每个像素点投影到三维空间，得到所述目标图像帧对应的三维点云。
S503：对所述目标图像帧对应的三维点云进行去噪处理。
本发明实施例中，点云融合设备可以对所述目标图像帧对应的三维点云进行去噪处理。
在一种实施方式中，本发明在进行去噪处理时，可以根据图像帧的位置信息如全球定位系统(Global Positioning System，GPS)信息或者通过空中三角测量得到的位置信息确定为目标图像帧，并将所述目标图像帧对应的三维点云投影到与所述目标图像帧相邻的图像帧的深度图上，以根据深度一致性进行去噪处理。在某些实施例中，整个投影过程是完全并行的，可以将图像帧的深度图细分成多个子区域并发，使用并行处理器加速以确保其实时性。
在一个实施例中，点云融合设备对所述目标图像帧对应的三维点云进行去噪处理时，可以获取与所述目标图像帧位置相邻的第三相邻图像帧的深度图和第四相邻图像帧的深度图，并根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理。
在一个实施例中，点云融合设备根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理时，可以将所述目标图像帧对应的三维点云投影到所述第三相邻图像帧的深度图上得到第一投影区域，并获取所述目标图像帧对应的三维点云与所述第一投影区域的第一深度差值；以及将所述目标图像帧对应的三维点云投影到所述第四相邻图像帧的深度图上得到第二投影区域，并获取所述目标图像帧对应的三维点 云与所述第二投影区域的第二深度差值；从而根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理。
在一个实施例中，点云融合设备根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理时，可以获取所述目标图像帧中所述第一深度差值和所述第二深度差值均小于第二预设阈值的目标三维点云，并删除所述目标图像帧中除所述目标三维点云以外的其余三维点云，以对所述目标图像帧对应的三维点云进行去噪处理。
具体可以图6为例对深度图进行去噪处理的过程进行说明，图6是本发明实施例提供的一种去噪处理的示意图，如图6所示，假设深度图62是当前图像帧对应的深度图，深度图61为目标图像帧，与所述目标图像帧61位置相邻的第三相邻图像帧的深度图为深度图60和深度图62。以一个像素点为例，目标图像帧对应的深度图61中的像素点u对应于三维空间中的三维点云p，将三维点云p投影到深度图60上可以得到一投影区域，具体可以为以投影点u为中心的第一投影区域，将三维点云p投影到深度图62上可以得到一投影区域，具体可以为以投影点u为中心的第二投影区域。比较像素点u的深度值与第一投影区域的深度值和第二投影区域的深度值，当第一投影区和第二投影区中包括深度值与像素点u点的深度值之差小于预定阈值，例如1％的像素点时，则可以保留深度图61中像素点u对应的三维点云p，反之则可以删除深度图61中像素点u对应的三维点云p，以完成去噪处理。可以理解的，当像素点u在相邻图像中没有对应的像素点与其的深度值之差在预定阈值范围内时，则认为像素点u的深度值是不可靠的，也即像素点u对应的三维点云的位置是不可靠的，通过这种去噪处理的方式，可以滤掉深度图中大部分的噪点，从而保证点云融合过程的准确性。
可见，通过这种对目标图像帧对应的三维点云进行去噪处理的方式，可以过滤无效三维点云，减少了噪点在融合处理过程中的开销，降低了内存占用率，有助于提高点云融合的有效性和效率。
S504：获取去噪处理后的与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云。
本发明实施例中，点云融合设备可以获取去噪处理后的与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云。具体实施例如前所述，此处不再 赘述。
S505：将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
本发明实施例中，点云融合设备可以将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。具体实施例如前所述，此处不再赘述。
本发明实施例，点云融合设备可以获取满足预设条件的目标图像帧对应的三维点云，并对所述目标图像帧对应的三维点云进行去噪处理，以及获取去噪处理后的与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。通过这种实施方式，可以减少噪点在融合处理过程中的开销，降低了内存占用率，有助于提高点云融合的有效性和效率。
具体请参见图7，图7是本发明实施例提供的又一种点云融合方法的流程示意图，所述方法可以由点云融合设备执行，其中，点云融合设备的具体解释如前所述。本发明实施例所述的方法与图5所述方法的区别在于，本发明实施例是对与当前图像帧相邻的前一图像帧的融合处理过程进行详细说明，具体地，本发明实施例的所述方法包括如下步骤。
S701：获取与当前图像帧相邻的前一图像帧的深度图。
本发明实施例中，点云融合设备可以获取与所述当前图像帧相邻的前一图像帧的深度图。
在一个实施例中，当点云融合设备获取到当前图像帧之前，所述点云融合设备可以获取与所述当前图像帧相邻的前一图像帧的深度图。
例如，假设当前图像帧为第12帧图像，与第12帧图像位置相邻的前一图像帧为第11帧图像，则点云融合设备可以获取与第12帧图像相邻的第11帧图像的深度图。
在一个实施例中，点云融合设备在获取与所述当前图像帧相邻的前一图像 帧的深度图之后，获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云之前，可以获取满足预设条件的目标图像帧对应的三维点云，并对所述目标图像帧对应的三维点云进行去噪处理。在某些实施例中，所述满足预设条件包括所述目标图像帧是所述前一图像帧位置相邻的前任一图像帧。
例如，假设当前图像帧为第12帧图像，当前图像帧相邻的前一图像帧为第11帧图像，则点云融合设备可以确定与第11帧图像位置相邻的第10帧图像为目标图像帧，并获取第10帧图像对应的三维点云，以及对所述第10帧图像对应的三维点云进行去噪处理。
S702：获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云。
本发明实施例中，点云融合设备可以获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云。
例如，假设当前图像帧为第12帧图像，当前图像帧相邻的前一图像帧为第11帧图像，则点云融合设备可以获取与第11帧图像位置相邻的第10帧图像对应的三维点云。
S703：将所述第二相邻图像帧对应的三维点云投影到所述前一图像帧的深度图上，以将所述第二相邻图像帧对应的三维点云与所述前一图像帧对应的三维点云进行融合处理。
本发明实施例中，点云融合设备可以将所述第二相邻图像帧对应的三维点云投影到所述前一图像帧的深度图上，以将所述第二相邻图像帧对应的三维点云与所述前一图像帧对应的三维点云进行融合处理。
例如，假设第二相邻图像帧为第10帧图像，当前图像帧为第12帧图像，与第12帧图像位置相邻的前一图像帧为第11帧图像，则点云融合设备可以将所述第10帧图像对应的三维点云与所述第11帧图像对应的三维点云进行融合处理。具体实施例如前所述，此处不再赘述。
在一个实施例中，图7示出的点云融合方法的流程为图2示出的点云融合方法的流程之前的循环之一。结合图1所示的应用场景，无人机在沿航线飞行的过程中，将拍摄到的图像帧实时发送至地面站，地面站接收到图像帧后根据这些图像帧实时生成三维点云。每当获取到新的图像帧时，新的图像帧中与之前获取的图像帧重叠的部分对应的三维点云需要和之前生成的三维点云进行 融合处理，而不重叠的部分则需要生成新的点云。如此，在无人机沿航线飞行的过程中，随着无人机拍摄到的图像帧越来越多，地面站实时生成的三维点云也越来越多，逐渐覆盖目标区域从而描绘出目标区域的三维信息。
请参见图8，图8是本发明实施例提供的一种点云融合设备的结构示意图。具体的，所述点云融合设备包括：存储器801、处理器802。
在一种实施例中，所述点云融合设备还包括数据接口803，所述数据接口803，用于传递点云融合设备和其他设备之间的数据信息。
所述存储器801可以包括易失性存储器(volatile memory)；存储器801也可以包括非易失性存储器(non-volatile memory)；存储器801还可以包括上述种类的存储器的组合。所述处理器802可以是中央处理器(central processing unit，CPU)。所述处理器802还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)或其任意组合。
所述存储器801用于存储程序，所述处理器802可以调用存储器801中存储的程序，用于执行如下步骤：
获取当前图像帧的深度图；
获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云；
将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
进一步地，所述处理器802获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云时，具体用于：
将所述第一相邻图像帧中的每个像素点投影到三维空间，得到所述第一相邻图像帧对应的三维点云。
进一步地，所述处理器802获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云时，具体用于：
获取所述第一相邻图像帧对应的三维点云的点云标记；
根据所述点云标记，获取所述第一相邻图像帧对应的三维点云。
进一步地，所述处理器802将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上时，具体用于：
获取所述当前图像帧对应的位置信息和姿态信息；
根据所述当前图像帧对应的位置信息和姿态信息，将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上。
进一步地，所述处理器802将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理之前，还用于：
获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域；
根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云；
所述处理器802将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理时，具体用于：
将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理。
进一步地，所述处理器802获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域时，具体用于：
获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影点；
以所述投影点为中心确定所述投影区域。
进一步地，所述处理器802根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云时，具体用于：
获取所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域中各个像素点的深度信息；
计算所述第一相邻图像帧对应的三维点云的深度信息与所述投影区域中各个像素点的深度信息的深度差值；
确定所述深度差值小于第一预设阈值的所述像素点对应对应的三维点云为有效三维点云。
进一步地，所述处理器802将所述第一相邻图像帧对应的三维点云与所述 当前图像帧的有效三维点云进行融合处理时，具体用于：
根据所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云生成融合点云；
其中，所述融合点云的深度信息是根据所述第一相邻图像帧对应的三维点云的深度信息与所述当前图像帧的有效三维点云的深度信息得到的。
进一步地，所述处理器802获取当前图像帧的深度图之后，获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云之前，还用于：
获取满足预设条件的目标图像帧对应的三维点云；
对所述目标图像帧对应的三维点云进行去噪处理。
进一步地，所述满足预设条件包括所述目标图像帧是所述当前图像帧位置相邻的前任一图像帧。
进一步地，所述处理器802获取满足预设条件的目标图像帧对应的三维点云时，具体用于：
将所述目标图像帧中的每个像素点投影到三维空间，得到所述目标图像帧对应的三维点云。
进一步地，所述处理器802对所述目标图像帧对应的三维点云进行去噪处理时，具体用于：
获取与所述目标图像帧位置相邻的第三相邻图像帧的深度图和第四相邻图像帧的深度图；
根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理。
进一步地，所述处理器802根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理时，具体用于：
将所述目标图像帧对应的三维点云投影到所述第三相邻图像帧的深度图上得到第一投影区域，并获取所述目标图像帧对应的三维点云与所述第一投影区域的第一深度差值；
将所述目标图像帧对应的三维点云投影到所述第四相邻图像帧的深度图上得到第二投影区域，并获取所述目标图像帧对应的三维点云与所述第二投影区域的第二深度差值；
根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理。
进一步地，所述处理器802根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理时，具体用于：
获取所述目标图像帧中所述第一深度差值和所述第二深度差值均小于第二预设阈值的目标三维点云；
删除所述目标图像帧中除所述目标三维点云以外的其余三维点云，以对所述目标图像帧对应的三维点云进行去噪处理。
进一步地，所述处理器802还用于：
获取与所述当前图像帧相邻的前一图像帧的深度图；
获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云；
将所述第二相邻图像帧对应的三维点云投影到所述前一图像帧的深度图上，以将所述第二相邻图像帧对应的三维点云与所述前一图像帧对应的三维点云进行融合处理。
进一步地，所述处理器802获取与所述当前图像帧相邻的前一图像帧的深度图之后，获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云之前，还用于：
获取满足预设条件的目标图像帧对应的三维点云；
对所述目标图像帧对应的三维点云进行去噪处理。
进一步地，所述满足预设条件包括所述目标图像帧是所述前一图像帧位置相邻的前任一图像帧。
本发明实施例中，点云融合设备通过获取当前图像帧的深度图，获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，并将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处。通过这种实施方式可以提高点云融合的效率和有效性、降低资源占用率。
本发明实施例还提供了一种点云融合系统，所述系统包括：可移动平台，所述可移动平台包括拍摄装置，用于对环境进行拍摄得到图像帧；以及上述点云融合设备。本发明实施例中，点云融合设备通过获取当前图像帧的深度图， 获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，并将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处。通过这种实施方式可以提高点云融合的效率和有效性、降低资源占用率。
本发明的实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现本发明图2、图5或图7所对应实施例中描述的方法，也可实现图8所述本发明所对应实施例的设备，在此不再赘述。
所述计算机可读存储介质可以是前述任一实施例所述的设备的内部存储单元，例如设备的硬盘或内存。所述计算机可读存储介质也可以是所述设备的外部存储设备，例如所述设备上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，所述计算机可读存储介质还可以既包括所述设备的内部存储单元也包括外部存储设备。所述计算机可读存储介质用于存储所述计算机程序以及所述终端所需的其他程序和数据。所述计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。
以上所揭露的仅为本发明部分实施例而已，当然不能以此来限定本发明之权利范围，因此依本发明权利要求所作的等同变化，仍属本发明所涵盖的范围。
Claims (36)
- 一种点云融合方法，其特征在于，包括：获取当前图像帧的深度图；获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云；将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
- 根据权利要求1所述的方法，其特征在于，所述获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，包括：将所述第一相邻图像帧中的每个像素点投影到三维空间，得到所述第一相邻图像帧对应的三维点云。
- 根据权利要求1所述的方法，其特征在于，所述获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云，包括：获取所述第一相邻图像帧对应的三维点云的点云标记；根据所述点云标记，获取所述第一相邻图像帧对应的三维点云。
- 根据权利要求1所述的方法，其特征在于，所述将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，包括：获取所述当前图像帧对应的位置信息和姿态信息；根据所述当前图像帧对应的位置信息和姿态信息，将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上。
- 根据权利要求4所述的方法，其特征在于，所述将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理之前，还包括：获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域；根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云；所述将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理，包括：将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理。
- 根据权利要求5所述的方法，其特征在于，所述获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域，包括：获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影点；以所述投影点为中心确定所述投影区域。
- 根据权利要求5所述的方法，其特征在于，所述根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云，包括：获取所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域中各个像素点的深度信息；计算所述第一相邻图像帧对应的三维点云的深度信息与所述投影区域中各个像素点的深度信息的深度差值；确定所述深度差值小于第一预设阈值的所述像素点对应对应的三维点云为有效三维点云。
- 根据权利要求5所述的方法，其特征在于，所述将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理，包括：根据所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云生成融合点云；其中，所述融合点云的深度信息是根据所述第一相邻图像帧对应的三维点云的深度信息与所述当前图像帧的有效三维点云的深度信息得到的。
- 根据权利要求1所述的方法，其特征在于，所述获取当前图像帧的深度图之后，获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云之前，还包括：获取满足预设条件的目标图像帧对应的三维点云；对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求9所述的方法，其特征在于，所述满足预设条件包括所述目标图像帧是所述当前图像帧位置相邻的前任一图像帧。
- 根据权利要求9所述的方法，其特征在于，所述获取满足预设条件的目标图像帧对应的三维点云，包括：将所述目标图像帧中的每个像素点投影到三维空间，得到所述目标图像帧对应的三维点云。
- 根据权利要求9所述的方法，其特征在于，所述对所述目标图像帧对应的三维点云进行去噪处理，包括：获取与所述目标图像帧位置相邻的第三相邻图像帧的深度图和第四相邻图像帧的深度图；根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求12所述的方法，其特征在于，所述根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理，包括：将所述目标图像帧对应的三维点云投影到所述第三相邻图像帧的深度图上得到第一投影区域，并获取所述目标图像帧对应的三维点云与所述第一投影区域的第一深度差值；将所述目标图像帧对应的三维点云投影到所述第四相邻图像帧的深度图 上得到第二投影区域，并获取所述目标图像帧对应的三维点云与所述第二投影区域的第二深度差值；根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求13所述的方法，其特征在于，所述根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理，包括：获取所述目标图像帧中所述第一深度差值和所述第二深度差值均小于第二预设阈值的目标三维点云；删除所述目标图像帧中除所述目标三维点云以外的其余三维点云，以对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求1所述的方法，其特征在于，所述方法还包括：获取与所述当前图像帧相邻的前一图像帧的深度图；获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云；将所述第二相邻图像帧对应的三维点云投影到所述前一图像帧的深度图上，以将所述第二相邻图像帧对应的三维点云与所述前一图像帧对应的三维点云进行融合处理。
- 根据权利要求15所述的方法，其特征在于，所述获取与所述当前图像帧相邻的前一图像帧的深度图之后，获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云之前，还包括：获取满足预设条件的目标图像帧对应的三维点云；对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求16所述的方法，其特征在于，所述满足预设条件包括所述目标图像帧是所述前一图像帧位置相邻的前任一图像帧。
- 一种点云融合设备，其特征在于，包括存储器和处理器；所述存储器，用于存储程序；所述处理器，用于调用所述程序，当所述程序被执行时，用于执行以下操作：获取当前图像帧的深度图；获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云；将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上，以将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理。
- 根据权利要求18所述的设备，其特征在于，所述处理器获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云时，具体用于：将所述第一相邻图像帧中的每个像素点投影到三维空间，得到所述第一相邻图像帧对应的三维点云。
- 根据权利要求18所述的设备，其特征在于，所述处理器获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云时，具体用于：获取所述第一相邻图像帧对应的三维点云的点云标记；根据所述点云标记，获取所述第一相邻图像帧对应的三维点云。
- 根据权利要求18所述的设备，其特征在于，所述处理器将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上时，具体用于：获取所述当前图像帧对应的位置信息和姿态信息；根据所述当前图像帧对应的位置信息和姿态信息，将所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上。
- 根据权利要求21所述的设备，其特征在于，所述处理器将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理之前，还用于：获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度 图上得到的投影区域；根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云；所述处理器将所述第一相邻图像帧对应的三维点云与所述当前图像帧对应的三维点云进行融合处理时，具体用于：将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理。
- 根据权利要求22所述的设备，其特征在于，所述处理器获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影区域时，具体用于：获取所述第一相邻图像帧对应的三维点云投影到所述当前图像帧的深度图上得到的投影点；以所述投影点为中心确定所述投影区域。
- 根据权利要求22所述的设备，其特征在于，所述处理器根据所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域的深度信息确定所述当前图像帧的有效三维点云时，具体用于：获取所述第一相邻图像帧对应的三维点云的深度信息和所述投影区域中各个像素点的深度信息；计算所述第一相邻图像帧对应的三维点云的深度信息与所述投影区域中各个像素点的深度信息的深度差值；确定所述深度差值小于第一预设阈值的所述像素点对应对应的三维点云为有效三维点云。
- 根据权利要求22所述的设备，其特征在于，所述处理器将所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云进行融合处理时，具体用于：根据所述第一相邻图像帧对应的三维点云与所述当前图像帧的有效三维点云生成融合点云；其中，所述融合点云的深度信息是根据所述第一相邻图像帧对应的三维点云的深度信息与所述当前图像帧的有效三维点云的深度信息得到的。
- 根据权利要求18所述的设备，其特征在于，所述处理器获取当前图像帧的深度图之后，获取与所述当前图像帧位置相邻的第一相邻图像帧对应的三维点云之前，还用于：获取满足预设条件的目标图像帧对应的三维点云；对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求26所述的设备，其特征在于，所述满足预设条件包括所述目标图像帧是所述当前图像帧位置相邻的前任一图像帧。
- 根据权利要求26所述的设备，其特征在于，所述处理器获取满足预设条件的目标图像帧对应的三维点云时，具体用于：将所述目标图像帧中的每个像素点投影到三维空间，得到所述目标图像帧对应的三维点云。
- 根据权利要求26所述的设备，其特征在于，所述处理器对所述目标图像帧对应的三维点云进行去噪处理时，具体用于：获取与所述目标图像帧位置相邻的第三相邻图像帧的深度图和第四相邻图像帧的深度图；根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求29所述的设备，其特征在于，所述处理器根据所述第三相邻图像帧的深度图和第四相邻图像帧的深度图，对所述目标图像帧对应的三维点云进行去噪处理时，具体用于：将所述目标图像帧对应的三维点云投影到所述第三相邻图像帧的深度图上得到第一投影区域，并获取所述目标图像帧对应的三维点云与所述第一投影 区域的第一深度差值；将所述目标图像帧对应的三维点云投影到所述第四相邻图像帧的深度图上得到第二投影区域，并获取所述目标图像帧对应的三维点云与所述第二投影区域的第二深度差值；根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求30所述的设备，其特征在于，所述处理器根据所述第一深度差值以及所述第二深度差值，对所述目标图像帧对应的三维点云进行去噪处理时，具体用于：获取所述目标图像帧中所述第一深度差值和所述第二深度差值均小于第二预设阈值的目标三维点云；删除所述目标图像帧中除所述目标三维点云以外的其余三维点云，以对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求18所述的设备，其特征在于，所述处理器还用于：获取与所述当前图像帧相邻的前一图像帧的深度图；获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云；将所述第二相邻图像帧对应的三维点云投影到所述前一图像帧的深度图上，以将所述第二相邻图像帧对应的三维点云与所述前一图像帧对应的三维点云进行融合处理。
- 根据权利要求32所述的设备，其特征在于，所述处理器获取与所述当前图像帧相邻的前一图像帧的深度图之后，获取与所述前一图像帧位置相邻的第二相邻图像帧对应的三维点云之前，还用于：获取满足预设条件的目标图像帧对应的三维点云；对所述目标图像帧对应的三维点云进行去噪处理。
- 根据权利要求33所述的设备，其特征在于，所述满足预设条件包括所述目标图像帧是所述前一图像帧位置相邻的前 任一图像帧。
- 一种点云融合系统，其特征在于，包括：可移动平台，所述可移动平台包括拍摄装置，用于对环境进行拍摄的得到图像帧；以及如权利要求18-34任一所述的点云融合设备。
- 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至17任一项所述方法。
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201980030384.2A CN112106112A (zh) | 2019-09-16 | 2019-09-16 | 一种点云融合方法、设备、系统及存储介质 |
| PCT/CN2019/105885 WO2021051220A1 (zh) | 2019-09-16 | 2019-09-16 | 一种点云融合方法、设备、系统及存储介质 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2019/105885 WO2021051220A1 (zh) | 2019-09-16 | 2019-09-16 | 一种点云融合方法、设备、系统及存储介质 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2021051220A1 true WO2021051220A1 (zh) | 2021-03-25 |
Family
ID=73748808
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2019/105885 WO2021051220A1 (zh) | 2019-09-16 | 2019-09-16 | 一种点云融合方法、设备、系统及存储介质 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112106112A (zh) |
| WO (1) | WO2021051220A1 (zh) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210264223A1 (en) * | 2020-02-25 | 2021-08-26 | Beijing Qingzhouzhihang Intelligent Technology Co., Ltd | Method and apparatus for asynchronous data fusion, storage medium and electronic device |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114630096B (zh) * | 2022-01-05 | 2023-10-27 | 深圳技术大学 | Tof相机点云的稠密化方法、装置、设备及可读存储介质 |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833786A (zh) * | 2010-04-06 | 2010-09-15 | 清华大学 | 三维模型的捕捉及重建方法和系统 |
| CN105374019A (zh) * | 2015-09-30 | 2016-03-02 | 华为技术有限公司 | 一种多深度图融合方法及装置 |
| CN107230225A (zh) * | 2017-04-25 | 2017-10-03 | 华为技术有限公司 | 三维重建的方法和装置 |
| US20180018805A1 (en) * | 2016-07-13 | 2018-01-18 | Intel Corporation | Three dimensional scene reconstruction based on contextual analysis |
| CN108665496A (zh) * | 2018-03-21 | 2018-10-16 | 浙江大学 | 一种基于深度学习的端到端的语义即时定位与建图方法 |
| CN109544677A (zh) * | 2018-10-30 | 2019-03-29 | 山东大学 | 基于深度图像关键帧的室内场景主结构重建方法及系统 |
-
2019
- 2019-09-16 WO PCT/CN2019/105885 patent/WO2021051220A1/zh active Application Filing
- 2019-09-16 CN CN201980030384.2A patent/CN112106112A/zh active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833786A (zh) * | 2010-04-06 | 2010-09-15 | 清华大学 | 三维模型的捕捉及重建方法和系统 |
| CN105374019A (zh) * | 2015-09-30 | 2016-03-02 | 华为技术有限公司 | 一种多深度图融合方法及装置 |
| US20180018805A1 (en) * | 2016-07-13 | 2018-01-18 | Intel Corporation | Three dimensional scene reconstruction based on contextual analysis |
| CN107230225A (zh) * | 2017-04-25 | 2017-10-03 | 华为技术有限公司 | 三维重建的方法和装置 |
| CN108665496A (zh) * | 2018-03-21 | 2018-10-16 | 浙江大学 | 一种基于深度学习的端到端的语义即时定位与建图方法 |
| CN109544677A (zh) * | 2018-10-30 | 2019-03-29 | 山东大学 | 基于深度图像关键帧的室内场景主结构重建方法及系统 |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210264223A1 (en) * | 2020-02-25 | 2021-08-26 | Beijing Qingzhouzhihang Intelligent Technology Co., Ltd | Method and apparatus for asynchronous data fusion, storage medium and electronic device |
| US11501123B2 (en) * | 2020-02-25 | 2022-11-15 | Beijing Qingzhouzhihang Intelligent Technology Co., Ltd | Method and apparatus for asynchronous data fusion, storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112106112A (zh) | 2020-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019161813A1 (zh) | 动态场景的三维重建方法以及装置和系统、服务器、介质 | |
| US20210141378A1 (en) | Imaging method and device, and unmanned aerial vehicle | |
| US9981742B2 (en) | Autonomous navigation method and system, and map modeling method and system | |
| CN110176032B (zh) | 一种三维重建方法及装置 | |
| WO2018086130A1 (zh) | 飞行轨迹的生成方法、控制装置及无人飞行器 | |
| CN111415409B (zh) | 一种基于倾斜摄影的建模方法、系统、设备和存储介质 | |
| WO2021035731A1 (zh) | 无人飞行器的控制方法、装置及计算机可读存储介质 | |
| CN111127524A (zh) | 一种轨迹跟踪与三维重建方法、系统及装置 | |
| CN110361005B (zh) | 定位方法、定位装置、可读存储介质及电子设备 | |
| US11315313B2 (en) | Methods, devices and computer program products for generating 3D models | |
| CN112634370A (zh) | 一种无人机打点方法、装置、设备及存储介质 | |
| KR20210104684A (ko) | 측량 및 매핑 시스템, 측량 및 매핑 방법, 장치 및 기기 | |
| CN114565863B (zh) | 无人机图像的正射影像实时生成方法、装置、介质及设备 | |
| WO2021004416A1 (zh) | 一种基于视觉信标建立信标地图的方法、装置 | |
| WO2020103023A1 (zh) | 一种测绘系统、测绘方法、装置、设备及介质 | |
| JP2015114954A (ja) | 撮影画像解析方法 | |
| WO2021185036A1 (zh) | 点云数据的生成及实时显示方法、装置、设备及介质 | |
| WO2024093635A1 (zh) | 相机位姿估计方法、装置及计算机可读存储介质 | |
| CN115035235A (zh) | 三维重建方法及装置 | |
| WO2021051220A1 (zh) | 一种点云融合方法、设备、系统及存储介质 | |
| WO2021046810A1 (zh) | 一种三维点云的实时显示方法、设备、系统及存储介质 | |
| WO2020237422A1 (zh) | 航测方法、飞行器及存储介质 | |
| US20220113421A1 (en) | Online point cloud processing of lidar and camera data | |
| WO2022077790A1 (en) | Representation data generation of three-dimensional mapping data | |
| CN113034347A (zh) | 倾斜摄影图像处理方法、装置、处理设备及存储介质 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19945931 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19945931 Country of ref document: EP Kind code of ref document: A1 |