WO2021185036A1 - 点云数据的生成及实时显示方法、装置、设备及介质 - Google Patents

点云数据的生成及实时显示方法、装置、设备及介质 Download PDF

Info

Publication number
WO2021185036A1
WO2021185036A1 PCT/CN2021/077689 CN2021077689W WO2021185036A1 WO 2021185036 A1 WO2021185036 A1 WO 2021185036A1 CN 2021077689 W CN2021077689 W CN 2021077689W WO 2021185036 A1 WO2021185036 A1 WO 2021185036A1
Authority
WO
WIPO (PCT)
Prior art keywords
image frame
point cloud
transformed
cloud data
image
Prior art date
Application number
PCT/CN2021/077689
Other languages
English (en)
French (fr)
Inventor
池鹏可
Original Assignee
广州极飞科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州极飞科技有限公司 filed Critical 广州极飞科技有限公司
Publication of WO2021185036A1 publication Critical patent/WO2021185036A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering

Definitions

  • the embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, device, and medium for generating and real-time display of point cloud data.
  • ground image maps can be quickly collected, fully automated three-dimensional modeling can be realized, and DSM (Digital Surface Model) and DOM (Digital Orthophoto Map) can be output. Digital forward image map) and other results.
  • DSM Digital Surface Model
  • DOM Digital Orthophoto Map
  • the image processing device needs to first obtain the ground image map under the set scene collected by the aerial drone, and generate a depth map corresponding to the ground image map, and generate the depth map based on the depth map and its position information. Point cloud data under the scene, and then generate the corresponding DSM and DOM.
  • the prior art has the following shortcomings: the existing point cloud data calculation method has a large amount of calculation and is complicated in calculation, and cannot dynamically generate a three-dimensional point cloud of the entire scene in real time.
  • the embodiments of the present application provide a method, device, equipment, and medium for generating and displaying point cloud data in real time, which can dynamically generate a three-dimensional point cloud of the entire scene in real time.
  • an embodiment of the present application provides a method for generating point cloud data, including:
  • point cloud data corresponding to the first image frame is calculated.
  • two adjacently shot image frames are acquired as the first image frame and the second image frame.
  • adjusting the first image frame and the second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition includes:
  • the first image frame and the second image frame are adjusted to the first transformed image frame and the second image frame that satisfy the binocular stereo matching condition. Transform the image frame.
  • the first image frame and the second image frame are image frames taken when the drone is flying along a set route
  • the determining that the first image frame and the second image frame are in the same straight line includes:
  • adjusting the first image frame and the second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition includes:
  • the calculating a first new camera projection matrix corresponding to the first image frame and a second new camera projection matrix corresponding to the second image frame according to the target rotation matrix includes:
  • a second new camera projection matrix P n2 corresponding to the second image frame is calculated.
  • the first new camera corresponding to the first image frame is calculated based on the camera internal parameters of the camera that took the first image frame, the target rotation matrix, and the image position of the first image frame
  • the projection matrix P n1 includes:
  • P n1 K[R n
  • P n2 K[R n
  • K is the camera internal parameter
  • R n is the target rotation matrix
  • C 1 is the image position of the first image frame
  • C 2 is the image position of the second image frame.
  • the calculating a first correction transformation matrix according to the first new camera projection matrix and calculating a second correction transformation matrix according to the second new camera projection matrix includes:
  • a second correction transformation matrix is calculated.
  • the calculating the first correction transformation matrix based on the camera internal parameters of the camera that took the first image frame, the first new camera projection matrix P n1, and the rotation matrix of the first image frame includes:
  • T 1 (P n1 *(K*R' 1 )')', calculate the first corrective transformation matrix T 1 ;
  • the calculating the second correction transformation matrix based on the camera internal parameters of the camera that took the second image frame, the second new camera projection matrix P n2, and the rotation matrix of the second image frame includes:
  • T 2 (P n2 *(K*R' 2 )')', calculate the second correction transformation matrix T 2 ;
  • the calculating a depth map corresponding to the first transformed image frame according to the disparity map includes:
  • the calculating the point cloud data corresponding to the first image frame according to the depth map and the first transformed image frame includes:
  • Point3d (X, Y, Z ) P 'n1 * (x, y, depth)
  • the first image frame is calculated corresponding point cloud data Point3d (X, Y, Z) ;
  • C 1 is the image position of the first image frame
  • C 2 is the image position of the second image frame
  • f is the focal length value of the camera internal parameter
  • disparity is the disparity value
  • baseline is the baseline
  • baseline
  • x and y are the horizontal coordinates of each pixel in the first transformed image frame.
  • the method further includes:
  • the mismatch point data in the depth map corresponding to the first transformed image frame is deleted.
  • adjusting the first image frame and the second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition includes:
  • the method includes:
  • the second image frame is regarded as a new first image frame, and the shooting time is after the second image frame.
  • a new second image frame is determined in the image frame to calculate the point cloud data corresponding to the new first image frame according to the new first image frame and the new second image frame.
  • the first image frame and the second image frame are image frames taken while the drone is flying; the first image frame and the second image frame are adjusted to the first image frame that satisfies the binocular stereo matching condition
  • the first transformed image frame and the second transformed image frame include:
  • the first image frame and the second image frame are transformed, so that the conjugate pixels in the first image frame and the second image frame
  • the determined pair of conjugate lines becomes collinear and parallel to the baseline between the first image frame and the second image frame.
  • an embodiment of the present application also provides a real-time display method of point cloud data, including:
  • a first image frame is acquired, and a second image frame adjacent to the first image frame is acquired, where;
  • the point cloud data corresponding to the first image frame is obtained by calculation, and the point cloud data is displayed in real time;
  • the operation of obtaining the second image frame adjacent to the first image frame is returned to perform until the processing of all the captured image frames is completely processed.
  • the real-time display method of point cloud data also includes:
  • the second image frame is regarded as the new first image frame, and then the second image frame adjacent to the first image frame is returned to perform the acquisition. Operation of image frames.
  • the first image frame is initialized as the first image collected by the drone.
  • an embodiment of the present application also provides a point cloud data generation device, including:
  • the image frame acquisition module is used to acquire the first image frame and the second image frame that are taken, and adjust the first image frame and the second image frame to the first transformed image frame and the second image frame that meet the binocular stereo matching condition.
  • Transform image frame
  • the depth map calculation module is configured to calculate a disparity map corresponding to the first transformed image frame according to the first transformed image frame and the second transformed image frame, and calculate the disparity map corresponding to the first transformed image frame according to the disparity map.
  • the point cloud data calculation module is configured to calculate the point cloud data corresponding to the first image frame according to the depth map and the first transformed image frame.
  • an embodiment of the present application also provides a real-time display device for point cloud data, including:
  • the image frame acquisition module is used to acquire the image frames captured by the drone in real time
  • the adjacent image frame determination module is configured to obtain a first image frame according to the shooting order of the image frames, and obtain a second image frame adjacent to the first image frame;
  • the point cloud data display module is configured to calculate the point cloud data corresponding to the first image frame according to the method described in the first aspect, and display the point cloud data in real time;
  • the return execution module is used to take the second image frame as the new first image frame, and then return to execute the operation of obtaining the second image frame adjacent to the first image frame until the complete control of all the image frames taken deal with.
  • an embodiment of the present application also provides a computer device, and the computer device includes:
  • One or more processors are One or more processors;
  • Storage device for storing one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the point cloud data generation method described in any of the first aspect.
  • an embodiment of the present application also provides a computer device, and the computer device includes:
  • One or more processors are One or more processors;
  • Storage device for storing one or more programs
  • the one or more processors implement the method for real-time display of point cloud data described in any of the second aspect.
  • an embodiment of the present application also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method for generating point cloud data described in any of the first aspect is implemented.
  • an embodiment of the present application also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method for real-time display of any point cloud data described in the second aspect is realized.
  • the embodiment of the present application adjusts the captured first image frame and the second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition, and according to the first transformed image frame and the second transformed image frame
  • the transformed image frame calculates the disparity map corresponding to the first transformed image frame, and calculates the depth map corresponding to the first transformed image frame based on the disparity map, so as to calculate the point cloud corresponding to the first image frame based on the depth map and the first transformed image frame Data, and display the point cloud data in real time, to solve the problem that existing point cloud data processing methods cannot dynamically generate and display the three-dimensional point cloud of the entire scene in real time, so as to dynamically generate and display the three-dimensional point cloud of the entire scene in real time.
  • FIG. 1 is one of the flowcharts of the method for generating point cloud data provided by an embodiment of the present application
  • FIG. 2 is the second flowchart of the point cloud data generation method provided by the embodiment of the present application.
  • Fig. 3 is a flowchart of a real-time display method of point cloud data provided by an embodiment of the present application
  • Fig. 4 is a schematic diagram of a point cloud data generating device provided by an embodiment of the present application.
  • Fig. 5 is a schematic diagram of a real-time display device for point cloud data provided by an embodiment of the present application
  • Fig. 6 is a schematic structural diagram of a computer device provided by an embodiment of the application.
  • Figure 1 is a flow chart of a method for generating point cloud data provided by an embodiment of the present application.
  • the method provided in this embodiment is applicable to real-time generation of point cloud data.
  • the method can be executed by a point cloud data generating device.
  • the apparatus can be implemented by software and/or hardware, and generally can be integrated in a computing device.
  • the method includes the following operations:
  • Binocular stereo matching (stereo matching) technology can simultaneously collect images from the left and right cameras with a certain position difference among the binocular cameras to obtain two images of the same scene from different angles, and then according to the stereo parallax relationship of the two images It can perform depth estimation and 3D digital scene reconstruction.
  • the two frames of images captured in a short period of time during the flight of the drone can be used as the binocular stereo matching.
  • the first image frame and the second image frame may be two image frames obtained by taking pictures with a camera device during the flight of the drone.
  • the two images may be two images with a relatively close shooting time.
  • the same object shot has a certain shooting angle gap.
  • the shooting time of the first image frame is earlier than the shooting time of the second image frame.
  • the first image frame and the second image frame can be performed according to preset binocular stereo matching conditions.
  • the adjustment processing is based on the first transformed image frame and the second transformed image frame obtained after the adjustment processing, and the subsequent point cloud data generation action is performed using the binocular stereo matching condition.
  • the binocular stereo matching condition may be a condition set for the first transformed image frame and the second transformed image frame for performing binocular stereo matching.
  • the first transformed image frame may be an image frame obtained by performing processing such as coplanar adjustment, image rotation, and resampling mapping transformation on the first image frame.
  • the second transformed image frame may be an image frame obtained by performing processing such as coplanar adjustment, image rotation, and resampling mapping transformation on the second image frame.
  • adjusting the first image frame and the second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition may include: if it is determined that the first If the image frame and the second image frame are on the same straight line, the first image frame and the second image frame are adjusted to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition.
  • the UAV can use the camera to take pictures in real time to obtain multiple continuous image frames.
  • the first image frame and the second image frame may be adjusted to satisfy the binocular stereo matching condition According to the first transformed image frame and the second transformed image frame, binocular stereo matching can be performed according to the first transformed image frame and the second transformed image frame.
  • the first image frame and the second image frame are image frames taken when the drone is flying along a set route; the first image frame and the second image frame are determined Being in the same straight line, may include: obtaining a first rotation matrix corresponding to the first image frame and a second rotation matrix corresponding to the second image frame; calculating the first offset corresponding to the first rotation matrix A flight angle, and a second yaw angle corresponding to the second rotation matrix; if the difference between the first yaw angle and the second yaw angle is within a preset difference range, then It is determined that the first image frame and the second image frame are in the same straight line.
  • the rotation matrix since the rotation matrix records the flight attitude data during the flight of the drone, the first rotation matrix and the second rotation matrix recorded with the flight attitude data can be used to calculate the first image frame of the drone when the first image frame is taken. A yaw angle and a second yaw angle when the second image frame was taken.
  • the set route may be a designated operation route for the drone, such as a route for controlling the drone to perform operations through a remote control, or a route for operation obtained through a control command sent to the drone, etc.
  • the embodiments of this application do not There is no restriction on the specific method of obtaining the set route.
  • the preset difference value can be set according to actual requirements, such as 2°, 3°, or 5°, etc.
  • the embodiment of the present application does not set the preset difference value range.
  • the first rotation matrix corresponding to the first image frame and the second rotation matrix corresponding to the second image frame may be acquired. Then the first yaw angle corresponding to the first rotation matrix is calculated according to the first rotation matrix, and the second yaw angle corresponding to the second rotation matrix is calculated according to the second rotation matrix. If the difference value between the first yaw angle and the second yaw angle is within the preset difference value range, it indicates that the first image frame and the second image frame are in the same straight line.
  • the first image frame and the second image frame may be two adjacent image frames, and the first image frame and the second image frame are adjusted to satisfy binocular stereo
  • the matching condition of the first transformed image frame and the second transformed image frame may include: if it is determined that the first image frame and the second image frame are on the same straight line, combining the first image frame and the second image frame The frames are adjusted to the first transformed image frame and the second transformed image frame that satisfy the binocular stereo matching condition. If it is determined that the first image frame and the second image frame are not on the same straight line, the second image frame is regarded as a new first image frame, and the shooting time is after the second image frame.
  • the posture of the first image frame image1 can be expressed as pose1(R1
  • the posture of the second image frame image2 can be expressed as pose2(R2
  • R1 is the first rotation matrix
  • C1 is the image position of the first image frame
  • R2 is the second rotation matrix
  • C2 is the image position of the second image frame.
  • the first yaw angle yaw1 of the drone body is calculated according to the first rotation matrix R1
  • the second yaw angle yaw2 of the drone body is calculated according to the second rotation matrix R2.
  • the second image frame can be used to update the first image frame, and the second image frame adjacent to the second image frame can be used to update the first image frame.
  • the three image frames are updated to the second image frame to continue processing until the processing of all image frames is completed.
  • the threshold can be 3°.
  • one or more image frames may be spaced between the first image frame and the second image frame.
  • the disparity map may be based on the first transformed image frame, the size of which is the size of the reference image, and the element value is the disparity value between the first transformed image frame and the second transformed image frame. image. According to the disparity map corresponding to the first transformed image frame, the depth map corresponding to the first transformed image frame can be calculated.
  • the point cloud data corresponding to the first image frame can be calculated according to the depth map and the first transformed image frame, so as to realize the real-time and dynamic generation of each image frame according to each image frame.
  • the point cloud data corresponding to each image frame that is, the point cloud data of the entire scene is incrementally generated.
  • the embodiment of the application adjusts the acquired first image frame and second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition, and according to the first transformed image frame and the second transformed image
  • the frame calculates the disparity map corresponding to the first transformed image frame, calculates the depth map corresponding to the first transformed image frame according to the disparity map, and calculates the point cloud data corresponding to the first image frame according to the depth map and the first transformed image frame, It solves the problem that the existing point cloud data generation method cannot dynamically generate the three-dimensional point cloud of the entire scene in real time, so as to dynamically generate the three-dimensional point cloud of the entire scene in real time.
  • FIG. 2 is a flowchart of another method for generating point cloud data according to an embodiment of the present application, which specifically includes adjusting the first image frame and the second image frame to a first transformed image frame that satisfies the binocular stereo matching condition And the specific implementation of the second transformed image frame.
  • the first image frame and the second image frame can be changed according to the data in the rotation matrix of the drone attitude recorded when the first image frame and the second image frame are collected, so that the first image frame and the second image frame can be changed.
  • the paired conjugate epipolar lines determined by the corresponding pixel points in the image frame and the second image frame become collinear and parallel to the baseline, thereby obtaining the first transformed image frame and the second transformed image frame.
  • the method may include:
  • S210 Acquire the first image frame and the second image frame that are taken, and adjust the first image frame and the second image frame to a first transformed image frame and a second transformed image frame that meet the binocular stereo matching condition.
  • S210 may specifically include:
  • S211 Calculate a target rotation matrix according to the first rotation matrix corresponding to the first image frame, the second rotation matrix corresponding to the second image frame, and the principle of epipolar geometry.
  • the epipolar geometry is the basic geometry of stereo imaging.
  • the target rotation matrix can be used to rotate the first image frame and the second image frame until they are coplanar and parallel to the baseline.
  • the baseline may be determined according to the absolute value of the image position difference between the second image frame and the first image frame.
  • the baseline may be the second image frame and the first image frame. The absolute value of the image position difference between.
  • the first new camera projection matrix may be recalculated according to the target rotation matrix
  • the second new camera projection matrix may be recalculated according to the target rotation matrix
  • the first new camera projection matrix corresponding to the first image frame and the second new camera projection matrix corresponding to the second image frame are calculated according to the target rotation matrix , May include: calculating a first new camera projection matrix corresponding to the first image frame based on the camera internal parameters of the camera that took the first image frame, the target rotation matrix, and the image position of the first image frame P n1 ; calculate a second new camera projection matrix P corresponding to the second image frame based on the camera internal parameters of the camera that took the second image frame, the target rotation matrix, and the image position of the second image frame n2 .
  • -R n C 1 ]; according to the formula: P n2 K[R n
  • K is the camera internal parameter
  • R n is the target rotation matrix
  • C 1 is the image position of the first image frame
  • C 2 is the image position of the second image frame
  • the first corrective transformation matrix is used to correct and transform the first image frame
  • the second corrective transformation matrix is used to correct and transform the second image frame.
  • Picture correction is the transformation of the spatial rectangular coordinate system, from one two-dimensional coordinate to another two-dimensional coordinate
  • the picture correction is a linear transformation, maintaining the "parallelism” and "flatness” of the image, that is, the original image in the image.
  • Straight line and parallel line, the original straight line and parallel line are still maintained after transformation.
  • the special transformations commonly used in mapping transformation include translation, scaling, flipping, rotation and shearing.
  • Correction transformation processing refers to the use of the three-point collinear condition of the perspective center, the image point and the target point to rotate the bearing surface (perspective surface) around the trace (perspective axis) by a certain angle according to the perspective rotation law, destroying the original projection
  • the beam of light can still maintain the constant transformation of the projection geometry on the shadow bearing surface.
  • the calculating a first correction transformation matrix according to the first new camera projection matrix, and calculating a second correction transformation matrix according to the second new camera projection matrix may include: The camera internal parameters of the camera of the first image frame, the first new camera projection matrix P n1 and the rotation matrix of the first image frame are calculated, and the first correction transformation matrix is calculated; the camera based on the camera that took the second image frame The internal parameters, the second new camera projection matrix P n2 and the rotation matrix of the second image frame are used to calculate a second correction transformation matrix.
  • K is the camera internal parameter
  • R 1 is the rotation matrix of the first image frame
  • R 2 is the rotation matrix of the second image frame.
  • the re-sampling mapping transformation refers to re-sampling the pixel point coordinates in the image frame according to a certain rule according to the correction transformation matrix, and performing the mapping transformation according to the pixel point coordinates obtained by the re-sampling through a preset function relationship.
  • the original camera projection matrix of the first image frame and the second image frame and the pixel position of each pixel in the image will change accordingly.
  • the first image frame needs to be resampled and transformed according to the first corrective transformation matrix to obtain the first transformed image frame.
  • the second image frame is resampled, mapped and transformed according to the second corrective transformation matrix to obtain the second transformed image frame.
  • the obtained first transformed image frame and the second transformed image frame satisfy the condition of binocular stereo matching.
  • calculating the disparity map corresponding to the first transformed image frame based on the first transformed image frame and the second transformed image frame may be: using a correlation matching algorithm to calculate the corresponding disparity map of the first transformed image frame Disparity map.
  • related matching algorithms include but are not limited to SGM (Semi-Global Matching, semi-global matching) algorithm, local stereo matching algorithm, global stereo matching algorithm, etc.
  • the calculating a depth map corresponding to the first transformed image frame according to the disparity map may include: according to a formula: Calculate the depth map depth corresponding to the first transformed image frame.
  • f is the focal length value of the camera's internal parameters
  • disparity is the disparity value
  • baseline is the baseline
  • baseline
  • the calculating the depth map corresponding to the first transformed image frame according to the disparity map may further include: comparing the depth map corresponding to the first transformed image frame with Consistency detection is performed on the adjacent preset number of depth maps; and the mismatch point data in the depth map corresponding to the first transformed image frame is deleted according to the consistency detection result.
  • the preset number can be set according to actual needs.
  • the preset number can be 10, 12, or 15, etc.
  • the embodiment of the present application does not limit the specific value of the preset number.
  • the depth map corresponding to the first transformed image frame can be checked for consistency with a preset number of adjacent depth maps; the first transformed image can be deleted according to the consistency detection result Mismatch point data in the depth map corresponding to the frame.
  • the first image frame corresponding to the first transformed image frame is the first frame image, there is no adjacent depth map.
  • the number of adjacent depth maps in the first transformed image frame is small, which cannot meet the requirement of consistency detection.
  • the preset number is required to be 12, and the number of adjacent depth maps in the first transformed image frame is 8. In the above case, consistency detection may not be performed on the depth map corresponding to the first transformed image frame.
  • the target rotation matrix is calculated according to the first rotation matrix corresponding to the first image frame, the second rotation matrix corresponding to the second image frame, and the principle of epipolar geometry.
  • the corresponding first new camera projection matrix, and the second new camera projection matrix corresponding to the second image frame respectively calculate the first correction transformation matrix and the second correction transformation matrix according to the first new camera projection matrix and the second new camera projection matrix , And then perform re-sampling mapping transformation on the first image frame and the second image frame according to the first corrective transformation matrix and the second corrective transformation matrix to obtain the first transformed image frame and the second transformed image frame.
  • the frame and the second transformed image frame calculate the point cloud data corresponding to the first transformed image frame, which can realize the real-time dynamic generation of a three-dimensional point cloud of the entire scene. And after the depth map corresponding to the first transformed image frame is calculated, the mismatched points in the depth map are deleted through consistency detection, so as to ensure the accuracy of the depth map data, thereby ensuring the accuracy of the point cloud data.
  • Fig. 3 is a flowchart of a real-time display method of point cloud data provided by an embodiment of the present application.
  • the method is applicable to the situation of real-time display of point cloud data.
  • the method can be executed by a real-time display device for point cloud data.
  • the device can be implemented by software and/or hardware, and can generally be integrated into a computer device.
  • the method includes the following operations:
  • the camera device can be used to take pictures in real time to obtain multiple image frames.
  • the first image frame in order to generate and display the point cloud data of the entire scene in real time, the first image frame may be obtained from the first image frame according to the shooting order of the image frames, and the adjacent image frame may be obtained.
  • the second image frame in order to generate and display the point cloud data of the entire scene in real time, the first image frame may be obtained from the first image frame according to the shooting order of the image frames, and the adjacent image frame may be obtained.
  • the method for real-time display of point cloud data may further include: if it is determined that the currently processed first image frame and the second image frame are not on the same straight line, the second image frame may be used as After the new first image frame, return to perform the operation of acquiring the second image frame adjacent to the first image frame.
  • the currently processed first image frame and the second image frame are not on the same straight line, which can be understood as the geometric correlation between the currently processed first image frame and the second image frame is weak, and it is difficult to be adjusted to conform to binocular stereo. The image that matches the criteria.
  • the processing can be continued.
  • it is determined that the currently processed first image frame and the second image frame are not on the same straight line after updating the second image frame to the first image frame, obtain a third image frame adjacent to the second image frame Update to the second image frame and continue processing.
  • S330 Calculate the point cloud data corresponding to the first image frame according to the point cloud data generation method provided in any of the foregoing embodiments, and display the point cloud data in real time.
  • the embodiment of the present application can start processing from the first frame of image, and generate and display the point cloud data of each frame of image in real time. It is understandable that whenever point cloud data is generated and displayed for a frame of image in real time, the point cloud data of the frame of image will be added in real time to the entire scene. Therefore, the real-time display method for point cloud data provided by the embodiments of the present application can be Realize the incremental display of the point cloud data of the entire scene.
  • the embodiment of the application displays the point cloud data in real time after generating the point cloud data of the scene in real time, which can solve the problem that the existing point cloud data generation method cannot display and generate the three-dimensional point cloud of the entire scene in real time, thereby real-time increment
  • the three-dimensional point cloud of the entire scene is displayed in a way.
  • the above-mentioned real-time display method of point cloud data can be integrated into the above-mentioned point cloud data generation method, and it can be understood that: the above-mentioned real-time display method of point cloud data can be used as the above-mentioned point cloud data
  • the steps included in the generation method for example, after the point cloud data is obtained by calculation, the method for generating point cloud data provided by any of the foregoing embodiments of the present application may further include: real-time display of the point cloud data.
  • the steps included in the real-time display method of point cloud data can be adaptively adjusted.
  • steps S310 and S320 can be deleted, and step S330 can be deleted.
  • the adaptive adjustment is: real-time display of the point cloud data, which can be understood as: in the point cloud data generation method, since the point cloud data has been calculated through steps S110, S120, and S130, it is shown that the point cloud data has been calculated.
  • the obtained point cloud data only needs to be displayed in real time after the point cloud data is obtained by calculation.
  • FIG. 4 is a schematic diagram of a point cloud data generation device provided by an embodiment of the present application. As shown in FIG. 4, the device includes: an image frame acquisition module 410, a depth map calculation module 420, and a point cloud data calculation module 430, in:
  • the image frame acquisition module 410 is configured to acquire the first image frame and the second image frame that are taken, and adjust the first image frame and the second image frame to the first transformed image frame and the second image frame that satisfy the binocular stereo matching condition. 2. Transform the image frame;
  • the depth map calculation module 420 is configured to calculate a disparity map corresponding to the first transformed image frame according to the first transformed image frame and the second transformed image frame, and calculate the disparity map corresponding to the first transformed image frame according to the disparity map. Transform the depth map corresponding to the image frame;
  • the point cloud data calculation module 430 is configured to calculate the point cloud data corresponding to the first image frame according to the depth map and the first transformed image frame.
  • the embodiment of the application adjusts the captured first image frame and the second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition, and according to the first transformed image frame and the second transformed image frame
  • the image frame calculates the disparity map corresponding to the first transformed image frame, and calculates the depth map corresponding to the first transformed image frame according to the disparity map, so as to calculate the point cloud data corresponding to the first image frame according to the depth map and the first transformed image frame , Solve the problem that the existing point cloud data generation method cannot dynamically generate the three-dimensional point cloud of the entire scene in real time, so as to dynamically generate the three-dimensional point cloud of the entire scene in real time.
  • the image frame acquisition module 410 includes a first image frame acquisition unit, configured to, if it is determined that the first image frame and the second image frame are on the same straight line, combine the first image frame and the second image frame The two image frames are adjusted to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition.
  • the first image frame and the second image frame are image frames taken when the drone is flying along a set route;
  • the first image frame acquiring unit is specifically configured to acquire and the first image frame The corresponding first rotation matrix, and the second rotation matrix corresponding to the second image frame; calculating the first yaw angle corresponding to the first rotation matrix, and the second rotation matrix corresponding to the second rotation matrix Yaw angle; if the difference value between the first yaw angle and the second yaw angle is within a preset difference value range, it is determined that the first image frame and the second image frame are in Same straight line.
  • the image frame acquisition module 410 may further include:
  • the target rotation matrix calculation unit is configured to calculate the target rotation matrix according to the first rotation matrix corresponding to the first image frame, the second rotation matrix corresponding to the second image frame, and the principle of epipolar geometry;
  • a new camera projection matrix calculation unit configured to calculate a first new camera projection matrix corresponding to the first image frame and a second new camera projection matrix corresponding to the second image frame according to the target rotation matrix;
  • a correction transformation matrix calculation unit configured to calculate a first correction transformation matrix according to the first new camera projection matrix, and calculate a second correction transformation matrix according to the second new camera projection matrix;
  • a first transformed image frame acquisition unit configured to perform re-sampling mapping transformation on the first image frame according to the first corrective transformation matrix to obtain a first transformed image frame
  • the second transformed image frame acquisition unit is configured to perform re-sampling mapping transformation on the second image frame according to the second corrective transformation matrix to obtain a second transformed image frame.
  • the new camera projection matrix calculation unit is specifically configured to calculate the relationship with the first image frame based on the camera internal parameters of the camera that took the first image frame, the target rotation matrix, and the image position of the first image frame.
  • the first new camera projection matrix P n1 corresponding to the image frame; based on the camera internal parameters of the camera that took the second image frame, the target rotation matrix, and the image position of the second image frame, the second image is calculated
  • the second new camera projection matrix P n2 corresponding to the frame.
  • P n1 K[R n
  • P n2 K[R n
  • the correction transformation matrix calculation unit is specifically configured to calculate the first correction transformation matrix based on the camera internal parameters of the camera that took the first image frame, the first new camera projection matrix P n1, and the rotation matrix of the first image frame Calculate a second correction transformation matrix based on the camera internal parameters of the camera that took the second image frame, the second new camera projection matrix P n2, and the rotation matrix of the second image frame.
  • T 1 (P n1 *(K*R' 1 )')'
  • T 2 (P n2 *(K*R' 2 )')'
  • the point cloud data generating device further includes: a consistency detection module, configured to perform consistency detection between the depth map corresponding to the first transformed image frame and a preset number of adjacent depth maps; mismatched point data
  • the deleting module is configured to delete the mismatch point data in the depth map corresponding to the first transformed image frame according to the consistency detection result.
  • the image frame obtaining module 410 may further include a second image frame obtaining unit, configured to use the second image if it is determined that the first image frame and the second image frame are not on the same straight line.
  • the first image frame is updated by the frame; and the operation of acquiring the captured first image frame and the second image frame is returned to execute until the processing of all the image frames is completed.
  • the above-mentioned point cloud data generation device can execute the point cloud data generation method provided by any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.
  • the point cloud data generation method provided in any embodiment of this application.
  • the point cloud data generating device introduced above is a device that can execute the point cloud data generating method in the embodiment of the present application, based on the point cloud data generating method introduced in the embodiment of the present application, the technology in the field Personnel can understand the specific implementation of the point cloud data generation device of this embodiment and its various changes, so here is how the point cloud data generation device implements the point cloud data generation method in the embodiment of this application. Let me introduce it in detail. As long as those skilled in the art implement the point cloud data generating method in the embodiments of the present application, the device shall fall within the scope of the protection of the present application.
  • FIG. 5 is a schematic diagram of a real-time display device for point cloud data provided by an embodiment of the present application. As shown in FIG. 5, the device includes: an image frame acquisition module 510, an adjacent image frame determination module 520, and a point cloud data display Module 530 and return execution module 540, where:
  • the image frame obtaining module 510 is used to obtain image frames obtained by real-time shooting by the drone;
  • the adjacent image frame determining module 520 is configured to obtain a first image frame according to the shooting order of the image frames, and obtain a second image frame adjacent to the first image frame;
  • the point cloud data display module 530 is configured to calculate the point cloud data corresponding to the first image frame according to the point cloud data generation method provided by any of the foregoing embodiments, and display the point cloud data in real time;
  • the return execution module 540 is configured to use the second image frame as the new first image frame, and then return to execute the operation of acquiring the second image frame adjacent to the first image frame until all the captured image frames are completely checked. Processing.
  • the device for real-time display of point cloud data may further include: an image frame update module, configured to: if it is determined that the currently processed first image frame and the second image frame are not on the same straight line, After being the new first image frame, return to perform the operation of acquiring the second image frame adjacent to the first image frame.
  • an image frame update module configured to: if it is determined that the currently processed first image frame and the second image frame are not on the same straight line, After being the new first image frame, return to perform the operation of acquiring the second image frame adjacent to the first image frame.
  • the embodiment of the application displays the point cloud data in real time after generating the point cloud data of the scene in real time, which can solve the problem that the existing point cloud data generation method cannot display and generate the three-dimensional point cloud of the entire scene in real time, thereby real-time increment
  • the three-dimensional point cloud of the entire scene is displayed in a way.
  • the above-mentioned real-time display device for point cloud data can execute the real-time display method for point cloud data provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.
  • the real-time display method of point cloud data provided by any embodiment of this application.
  • the real-time display device for point cloud data introduced above is a device that can execute the real-time display method for point cloud data in the embodiment of this application, based on the real-time display method for point cloud data introduced in the embodiment of this application, this Those skilled in the art can understand the specific implementation of the point cloud data real-time display device of this embodiment and its various changes, so here is how the point cloud data real-time display device implements the point cloud in the embodiment of this application
  • the real-time display method of data will not be introduced in detail. As long as those skilled in the art implement the method for real-time display of point cloud data in the embodiments of this application, all devices fall within the scope of this application.
  • the above-mentioned real-time display device for point cloud data can be integrated into the above-mentioned point cloud data generating device, and it can be understood that: the above-mentioned real-time display device for point cloud data can be used as the point cloud data
  • the modules included in the generating device, for example, the point cloud data generating device provided by any of the foregoing embodiments of the present application may further include: the point cloud data display module and the return execution module.
  • the functions of the modules included in the point cloud data real-time display device can be adjusted adaptively.
  • the point cloud data display module can directly display
  • the point cloud data calculated by the point cloud data calculation module can be understood as: the point cloud data display module does not need to calculate the point cloud data, and is only responsible for displaying the point cloud data.
  • FIG. 6 is a schematic structural diagram of a computer device provided by an embodiment of this application.
  • FIG. 6 shows a block diagram of a computer device 612 suitable for implementing the embodiments of the present application.
  • the computer device 612 shown in FIG. 6 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present application.
  • the computer device 612 is represented in the form of a general-purpose computing device.
  • the components of the computer device 612 may include, but are not limited to: one or more processors 616, a storage device 628, and a bus 618 connecting different system components (including the storage device 628 and the processor 616).
  • the bus 618 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure among multiple bus structures.
  • these architectures include but are not limited to Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, enhanced ISA bus, Video Electronics Standards Association (Video Electronics Standards Association) , VESA) local bus and Peripheral Component Interconnect (PCI) bus.
  • Computer device 612 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 612, including volatile and nonvolatile media, removable and non-removable media.
  • the storage device 628 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) 630 and/or a cache memory 632.
  • the computer device 612 may further include other removable/non-removable, volatile/nonvolatile computer system storage media.
  • the storage system 634 may be used to read and write non-removable, non-volatile magnetic media (not shown in FIG. 6 and generally referred to as a "hard drive").
  • a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and a removable non-volatile optical disk (such as a compact disc) can be provided.
  • each drive can be connected to the bus 618 through one or more data media interfaces.
  • the storage device 628 may include at least one program product, and the program product has a set of (for example, at least one) program modules, and these program modules are configured to perform the functions of the embodiments of the present application.
  • a program 636 having a set of (at least one) program module 626 may be stored in, for example, the storage device 628.
  • Such program module 626 includes but is not limited to an operating system, one or more application programs, other program modules, and program data. Each of the examples or some combination may include the realization of a network environment.
  • the program module 626 generally executes the functions and/or methods in the embodiments described in this application.
  • the computer device 612 can also communicate with one or more external devices 614 (such as keyboards, pointing devices, cameras, displays 624, etc.), and can also communicate with one or more devices that enable users to interact with the computer device 612, and/ Or communicate with any device (such as a network card, modem, etc.) that enables the computer device 612 to communicate with one or more other computing devices. This communication can be performed through an input/output (Input/Output, I/O) interface 622.
  • the computer device 612 may also communicate with one or more networks (such as a local area network (LAN), a wide area network, WAN) and/or a public network, such as the Internet, through the network adapter 620. As shown in FIG.
  • the network adapter 620 communicates with other modules of the computer device 612 through the bus 618. It should be understood that although not shown in the figure, other hardware and/or software modules can be used in conjunction with the computer device 612, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, and disk arrays (Redundant Arrays). of Independent Disks, RAID) systems, tape drives, and data backup storage systems.
  • the processor 616 executes various functional applications and data processing by running programs stored in the storage device 628, such as implementing the point cloud data generation method provided in the foregoing embodiments of the present application.
  • the processing unit executes the program, it realizes: acquiring the first image frame and the second image frame that were taken, and adjusting the first image frame and the second image frame to the first image that satisfies the binocular stereo matching condition.
  • a transformed image frame and a second transformed image frame according to the first transformed image frame and the second transformed image frame, a disparity map corresponding to the first transformed image frame is calculated, and based on the disparity map, the The depth map corresponding to the first transformed image frame; and the point cloud data corresponding to the first image frame is calculated according to the depth map and the first transformed image frame.
  • the embodiment of the application also provides a computer device for executing the real-time display method of point cloud data provided by the implementation of the application.
  • the device includes: one or more processors; a storage device for storing one or more Programs; when the one or more programs are executed by the one or more processors, the one or more processors realize the real-time display method of point cloud data as provided in the embodiments of the present application: get unmanned The image frames obtained by real-time shooting by the machine; according to the shooting order of the image frames, the first image frame is obtained, and the second image frame adjacent to the first image frame is obtained, wherein the first image frame is initialized as a drone acquisition The first frame of image; according to the point cloud data generation method provided by any embodiment of this application, the point cloud data corresponding to the first image frame is calculated, and the point cloud data is displayed in real time; the first After the second image frame is used as the new first image frame, the operation of acquiring the second image frame adjacent to the first image frame is returned to execute until the processing of all the captured image frames is
  • An embodiment of the present application also provides a computer storage medium storing a computer program, when the computer program is executed by a computer processor, it is used to execute the point cloud data generation method described in any of the foregoing embodiments of the present application:
  • the first image frame and the second image frame, and the first image frame and the second image frame are adjusted to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition; according to the first transformation Calculating the disparity map corresponding to the first transformed image frame for the image frame and the second transformed image frame, and calculating the depth map corresponding to the first transformed image frame according to the disparity map; according to the depth map and
  • the first transformed image frame calculates point cloud data corresponding to the first image frame.
  • the computer storage medium of the embodiment of the present application may adopt any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above.
  • computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (Read Only Memory) , ROM), Erasable Programmable Read Only Memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or the above Any suitable combination of.
  • the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • suitable medium including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • the computer program code used to perform the operations of this application can be written in one or more programming languages or a combination thereof.
  • the programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect to the user's computer) connect).
  • LAN local area network
  • WAN wide area network
  • the embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by the processor, the real-time display method of point cloud data provided by the above-mentioned embodiments can be realized: obtaining real-time drones Image frames obtained by shooting; according to the order in which the image frames are taken, the first image frame is obtained, and the second image frame adjacent to the first image frame is obtained, wherein the first image frame is initialized as the first image collected by the drone Frame image; according to the point cloud data generation method provided by any of the above-mentioned embodiments of this application, the point cloud data corresponding to the first image frame is calculated, and the point cloud data is displayed in real time; the second After the image frame is taken as the new first image frame, the operation of acquiring the second image frame adjacent to the first image frame is returned to be executed until the processing of all the captured image frames is completely processed.
  • the embodiment of the present application adjusts the captured first image frame and second image frame to the first transformed image frame and the second transformed image frame that meet the binocular stereo matching condition, and according to the first transformed image frame and the second transformed image
  • the frame calculates the disparity map corresponding to the first transformed image frame, calculates the depth map corresponding to the first transformed image frame according to the disparity map, and calculates the point cloud data corresponding to the first image frame according to the depth map and the first transformed image frame
  • the point cloud data is displayed in real time, which solves the problem that existing point cloud data processing methods cannot dynamically generate and display the three-dimensional point cloud of the entire scene in real time, so as to dynamically generate and display the three-dimensional point cloud of the entire scene in real time.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例公开了一种点云数据的生成及实时显示方法、装置、设备及介质，方法包括：获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。本申请实施例的技术方案能够实时动态地生成整个场景的三维点云。

Description

点云数据的生成及实时显示方法、装置、设备及介质
相关申请的交叉引用
本申请要求于2020年03月18日提交中国专利局的申请号为2020101911553、名称为“点云数据的生成及实时显示方法、装置、设备及介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，尤其涉及一种点云数据的生成及实时显示方法、装置、设备及介质。
背景技术
随着无人机拍摄技术的不断发展，借助航拍无人机，可快速采集地面影像图，实现全自动化的三维建模，输出DSM(Digital Surface Model，数字表面模型)以及DOM(Digital Orthophoto Map，数字正向影像图)等成果。
现有技术中，图像处理设备需要首先获取与航拍无人机所采集的设定场景下的地面影像图，并生成与地面影像图对应的深度图，基于该深度图以及其位置信息，生成该场景下的点云数据，并进而生成相应的DSM以及DOM。
然而，现有技术存在如下缺陷：现有的点云数据的计算方法计算量大、计算复杂，无法实时动态地生成整个场景的三维点云。
发明内容
本申请实施例提供一种点云数据的生成及实时显示方法、装置、设备及介质，可以实时动态地生成整个场景的三维点云。
第一方面，本申请实施例提供了一种点云数据的生成方法，包括：
获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；
根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；
根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
可选的，获取相邻拍摄的两个图像帧作为所述第一图像帧和第二图像帧。
可选的，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
如果确定所述第一图像帧与所述第二图像帧处于同一直线上，则将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。
可选的，所述第一图像帧和第二图像帧为无人机沿设定航线飞行时所拍摄得到的图像帧；
确定所述第一图像帧与所述第二图像帧处于同一直线包括：
获取与所述第一图像帧对应的第一旋转矩阵，以及与所述第二图像帧对应的第二旋转矩阵；
计算与所述第一旋转矩阵对应的第一偏航角，以及与所述第二旋转矩阵对应的第二偏航角；
如果所述第一偏航角与所述第二偏航角之间的差异值在预设的差异值范围内，则确定所述第一图像帧与所述第二图像帧处于同一直线。可选的，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
根据所述第一图像帧对应的第一旋转矩阵、所述第二图像帧对应的第二旋转矩阵及对极几何原理，计算目标旋转矩阵；
根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵；
根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵；
根据所述第一矫正变换矩阵对所述第一图像帧进行重采样映射变换，得到第一变换图像帧；
根据所述第二矫正变换矩阵对所述第二图像帧进行重采样映射变换，得到第二变换图像帧。
可选的，所述根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵，包括：
基于拍摄所述第一图像帧的相机的相机内参、所述目标旋转矩阵和所述第一图像帧的图像位置，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；
基于拍摄所述第二图像帧的相机的相机内参、所述目标旋转矩阵和所述第二图像帧的图像位置，计算与所述第二图像帧对应的第二新相机投影矩阵P n2。
可选的，所述基于拍摄所述第一图像帧的相机的相机内参、所述目标旋转矩阵和所述第一图像帧的图像位置，计算与所述第一图像帧对应的第一新相机投影矩阵P n1，包括：
根据公式：P n1＝K[R n|-R nC 1]，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；
所述基于拍摄所述第二图像帧的相机的相机内参、所述目标旋转矩阵和所述第二图像帧的图像位置，计算与所述第二图像帧对应的第二新相机投影矩阵P n2，包括：
根据公式：P n2＝K[R n|-R nC 2]，计算与所述第一图像帧对应的第一新相机投影矩阵P n2；
其中，K为相机内参，R n为所述目标旋转矩阵，C 1为所述第一图像帧的图像位置，C 2为所述第二图像帧的图像位置。
可选的，所述根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵，包括：
基于拍摄所述第一图像帧的相机的相机内参、所述第一新相机投影矩阵P n1和所述第一图像帧的旋转矩阵，计算第一矫正变换矩阵；
基于拍摄所述第二图像帧的相机的相机内参、所述第二新相机投影矩阵P n2和所述第二图像帧的旋转矩阵，计算第二矫正变换矩阵。
可选地，所述基于拍摄所述第一图像帧的相机的相机内参、所述第一新相机投影矩阵P n1和所述第一图像帧的旋转矩阵，计算第一矫正变换矩阵，包括：
根据公式：T 1＝(P n1*(K*R' 1)')'，计算与所述第一矫正变换矩阵T 1；
可选地，所述基于拍摄所述第二图像帧的相机的相机内参、所述第二新相机投影矩阵P n2和所述第二图像帧的旋转矩阵，计算第二矫正变换矩阵，包括：
根据公式：T 2＝(P n2*(K*R' 2)')'，计算与所述第二矫正变换矩阵T 2；
可选地，所述根据所述视差图，计算与所述第一变换图像帧对应的深度图，包括：
根据公式：
Figure PCTCN2021077689-appb-000001
计算与所述第一变换图像帧对应的深度图depth；
所述根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据，包括：
根据公式：Point3d(X,Y,Z)＝P' n1*(x,y,depth)，计算所述第一图像帧对应的点云数据Point3d(X,Y,Z)；
其中，C 1为所述第一图像帧的图像位置，C 2为所述第二图像帧的图像位置，f为相机内参的焦距值，disparity为视差值，baseline为基线，baseline＝|C 2-C 1|，x、y为所述第一变换图像帧中各像素点的水平坐标。
可选的，在所述根据所述视差图，计算与所述第一变换图像帧对应的深度图之后，还包括：
将所述第一变换图像帧对应的深度图与相邻预设数量的深度图进行一致性检测；
根据一致性检测结果删除所述第一变换图像帧对应的深度图中的误匹配点数据。
可选的，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
如果确定所述第一图像帧与所述第二图像帧未处于同一直线上，则采用所述第二图像帧更新所述第一图像帧；
返回执行获取相邻拍摄的第一图像帧和第二图像帧的操作，直至完成对全部图像帧的处理。
可选地，所述方法，包括：
如果确定所述第一图像帧与所述第二图像帧未处于同一直线上，则将所述第二图像帧作为新的第一图像帧，并于拍摄时间在所述第二图像帧之后的图像帧中确定新的第二图像帧，以根据所述新的第一图像帧和新的第二图像帧计算与所述新的第一图像帧对应的点云数据。
可选地，所述第一图像帧和第二图像帧为无人机飞行时所拍摄得到的图像帧；将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
获取拍摄所述第一图像帧时所述无人机的第一旋转矩阵，以及拍摄所述第二图像帧时所述无人机的第二旋转矩阵；
根据所述第一旋转矩阵及所述第二旋转矩阵，对所述第一图像帧和所述第二图像帧进行变换处理，使由所述第一图像帧和第二图像帧中共轭像素点确定的成对共轭线变为共线且与所述第一图像帧和所述第二图像帧之间的基线平行。
第二方面，本申请实施例还提供了一种点云数据的实时显示方法，包括：
获取无人机实时拍摄得到的图像帧；
按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧，其中；
按照如第一方面所述的方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；
将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
可选的，点云数据的实时显示方法还包括：
如果确定当前处理的第一图像帧与第二图像帧未处于同一直线上，则将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作。
可选地，所述第一图像帧初始化为无人机采集的首帧图像。
第三方面，本申请实施例还提供了一种点云数据的生成装置，包括：
图像帧获取模块，用于获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；
深度图计算模块，用于根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；
点云数据计算模块，用于根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
第四方面，本申请实施例还提供了一种点云数据的实时显示装置，包括：
图像帧获取模块，用于获取无人机实时拍摄得到的图像帧；
相邻图像帧确定模块，用于按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧；
点云数据显示模块，用于按照如第一方面所述的方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；
返回执行模块，用于将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
第五方面，本申请实施例还提供了一种计算机设备，所述计算机设备包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现第一方面中任意所述的点云数据的生成方法。
第六方面，本申请实施例还提供了一种计算机设备，所述计算机设备包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现第二方面中任意所述的点云数据的实时显示方法。
第七方面，本申请实施例还提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现第一方面中任意所述的点云数据的生成方法。
第八方面，本申请实施例还提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现第二方面中任意所述的点云数据的实时显示方法。
本申请实施例通过对获取的拍摄的第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，并根据第一变换图像帧和第二变换图像帧计算与第一变换图像帧对应的视差图，根据视差图计算与第一变换图像帧对应的深度图，以根据深度图以及第一变换图像帧计算与第一图像帧对应的点云数据，并实时显示该点云数据，解决现有点云数据处理方法存在的无法实时动态地生成并显示整个场景的三维点云的问题，从而实时动态地生成并显示整个场景的三维点云。
附图说明
图1是本申请实施例提供的点云数据的生成方法的流程图之一；
图2是本申请实施例提供的点云数据的生成方法的流程图之二；
图3是本申请实施例提供的点云数据的实时显示方法的流程图；
图4是本申请实施例提供的点云数据的生成装置的示意图；
图5是本申请实施例提供的点云数据的实时显示装置的示意图；
图6为本申请实施例提供的计算机设备的结构示意图。
具体实施方式
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。
另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部内容。在更加详细地讨论示例性实施例之前应当提到的是，一些示例性实施例被描述成作为流程图描绘的处理或方法。虽然流程图将各项操作(或步骤)描述成顺序的处理，但是其中的许多操作可以被并行地、并发地或者同时实施。此外，各项操作的顺序可以被重新安排。当其操作完成时所述处理可以被终止，但是还可以具有未包括在附图中的附加步骤。所述处理可以对应于方法、函数、规程、子例程、子程序等等。
本申请实施例的说明书和权利要求书及附图中的术语“第一”和“第二”等是用于区别不同的对象，而不是用于描述特定的顺序。此外术语“包括”和“具有”以及他们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有设定于已列出的步骤或单元，而是可包括没有列出的步骤或单元。
图1是本申请实施例提供的一种点云数据的生成方法的流程图，本实施例提供的方法可适用于实时生成点云数据，该方法可以由点云数据的生成装置来执行，该装置可以由软件和/或硬件的方式来实现，并一般可集成在计算设备中。相应的，如图1所示，该方法包括如下操作：
S110、获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。
双目立体匹配(stereo matching)技术可以通过双目摄像机中具有一定位置差异的左右两个摄像机同时进行图像采集，获得从不同角度拍摄同一场景的两个图像，然后根据两个图像的立体视差关系可以进行深度估计、3D数字场景重建等工作。
在本实施例中，由于地表的高度数据通常在短时间内不会发生较大变化，因此可以将无人机飞行过程中短时间内先后拍摄到的两帧图像作为双目立体匹配中使用的两副图像，从而可以利用双目立体匹配技术进行地面物体的深度估计、3D数字场景重建等工作。
第一图像帧和第二图像帧可以是无人机在飞行过程中，利用摄像装置拍照获取的两帧图像帧，该两帧图像可以为拍摄时间较为接近的两帧图像，两帧图像中所拍摄到的同一物体具有一定的拍摄角度差距。可选的，第一图像帧的拍摄时间要比第二图像帧的拍摄时间靠前。
在双目立体匹配中，通常要求两个图像具有一定几何参考基准或几何相关性，因此在本实施例中可以根据预设的双目立体匹配条件对第一图像帧及第二图像帧进行一些调整处理，然后基于调整处理后获得第一变换图像帧和第二变换图像帧，利用双目立体匹配条件进行后续的点云数据生成动作。双目立体匹配条件可以是对第一变换图像帧和第二变换图像帧设定的，用于进行双目立体匹配的条件。第一变换图像帧可以是对第一图像帧进行诸如共面调整、图像旋转及重采样映射变换等处理后，得到的图像帧。相应的，第二变换图像帧可以是对第二图像帧进行诸如共面调整、图像旋转及重采样映射变换等处理后，得到的图像帧。
在一个可选实施方式中，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，可以包括：如果确定所述第一图像帧与所述第二图像帧处于同一直线上，则将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。
无人机在飞行过程中，可以利用摄像装置实时拍照，从而得到多个连续的图像帧。在本申请实施例中，为了实时生成点云数据，可以在确定第一图像帧与第二图像帧处于同一直线上时，将第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，以根据第一变换图像帧和第二变换图像帧，可以进行双目立体匹配。
在一个可选实施方式中，所述第一图像帧和第二图像帧为无人机沿设定航线飞行时所拍摄得到的图像帧；确定所述第一图像帧与所述第二图像帧处于同一直线，可以包括：获取与所述第一图像帧对应的第一旋转矩阵，以及与所述第二图像帧对应的第二旋转矩阵；计算与所述第一旋转矩阵对应的第一偏航角，以及与所述第二旋转矩阵对应的第二偏航角；如果所述第一偏航角与所述第二偏航角之间的差异值在预设的差异值范围内，则确定所述第一图像帧与所述第二图像帧处于同一直线。其中，由于旋转矩阵记录有无人机飞行过程中的飞行姿态数据，因此可以通过记录有飞行姿态数据的第一旋转矩阵和第二旋转矩阵计算出无人机在拍摄第一图像帧时的第一偏航角和在拍摄第二图像帧时的第二偏航角。
其中，设定航线可以是对无人机指定的作业航线，如通过遥控器控制无人机进行作业的航线，或通过对无人机发送的控制指令得到的作业航线等，本申请实施例并不对设定航线的具体获取方式进行限定。预设的差异值可以根据实际需求设定，如2°、3°或5°等，本申请实施例并不对预设的差异值范围进行设定。
具体的，在确定第一图像帧与第二图像帧是否处于同一直线时，可以获取与第一图像帧对应的第一旋转矩阵，以及与第二图像帧对应的第二旋转矩阵。然后根据第一旋转矩阵计算与第一旋转矩阵对应的第一偏航角，根据第二旋转矩阵计算与第二旋转矩阵对 应的第二偏航角。如果第一偏航角与第二偏航角之间的差异值在预设的差异值范围内，则表明第一图像帧与第二图像帧处于同一直线。
在一个可选的实现方式中，所述第一图像帧和所述第二图像帧可以为相邻的两个图像帧，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，可以包括：如果确定所述第一图像帧与所述第二图像帧处于同一直线上，则将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。如果确定所述第一图像帧与所述第二图像帧未处于同一直线上，则将所述第二图像帧作为新的第一图像帧，并于拍摄时间在所述第二图像帧之后的图像帧中确定新的第二图像帧，以根据所述新的第一图像帧和新的第二图像帧计算与所述新的第一图像帧对应的点云数据；返回执行获取相邻拍摄的第一图像帧和第二图像帧的操作，直至完成对全部图像帧的处理。
在一个具体的例子中，假设在无人机飞行过程中，利用相机实时拍照，并通过SLAM(Simultaneous Localization And Mapping，即时定位与地图构建)算法即时恢复当前帧图像的姿态。在相邻两帧图像中，第一图像帧image1的姿态可以表示为pose1(R1|C1)，第二图像帧image2的姿态可以表示为pose2(R2|C2)。其中，R1为第一旋转矩阵，C1为第一图像帧的图像位置；R2为第二旋转矩阵，C2为第二图像帧的图像位置。根据第一旋转矩阵R1计算无人机机体的第一偏航角yaw1，根据第二旋转矩阵R2计算无人机机体的第二偏航角yaw2。计算第一偏航角yaw1与第二偏航角yaw2之间的差异值△yaw。如果△yaw<threshold，则表明第一图像帧与第二图像帧处于同一直线。相应的，如果△yaw>threshold，则表明第一图像帧与第二图像帧未处于同一直线，此时可以采用第二图像帧更新第一图像帧，并将与第二图像帧相邻的第三图像帧更新为第二图像帧继续处理，直到完成对全部图像帧的处理。可选的，threshold可以为3°。
需要说明的是，上述获取相邻的图像帧作为第一图像帧和第二图像帧是为了保证所述第一图像帧中拍摄地面场景与所述第二图像帧中拍摄的地面场景大致一致，且针对拍摄到的同一物体，第一图像帧和第二图像帧存在一定的视角差。因此，只要能够满足上述要求，在其他实施方式中，所述第一图像帧和第二图像帧之间也可以间隔一个或多个图像帧。
S120、根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图。
在本申请实施例中，视差图可以是以第一变换图像帧为基准，其大小为该基准图像的大小，元素值为第一变换图像帧和第二变换图像帧之间的视差值的图像。根据第一变换图像帧对应的视差图，可以计算与第一变换图像帧对应的深度图。
S130、根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
相应的，得到第一变换图像帧对应的深度图后，即可根据深度图以及第一变换图像帧计算与第一图像帧对应的点云数据，从而实现根据每个图像帧实时动态地生成每个图像帧对应的点云数据，也即增量式生成整个场景的点云数据。
本申请实施例通过将获取的第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，并根据第一变换图像帧和第二变换图像帧计算与第一变换图像帧对应的视差图，根据视差图计算与第一变换图像帧对应的深度图，以根据深度图以及第一变换图像帧计算与第一图像帧对应的点云数据，解决现有点云数据生成方法存在的无法实时动态地生成整个场景的三维点云的问题，从而实时动态地生成整个场景的三维点云。
图2是本申请实施例提供的另一种点云数据的生成方法的流程图，具体包括将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧的具体实现方式。在本实施例中，可以根据采集第一图像帧和第二图像帧时所记录 的无人机姿态的旋转矩阵中的数据，对第一图像帧和第二图像帧进行变化处理，使第一图像帧和第二图像帧中由相对应的像素点确定的成对的共轭对极线变为共线且与基线平行，从而获得第一变换图像帧及第二变换图像帧。相应的，如图2所示，该方法可以包括：
S210、获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。
相应的，S210具体可以包括：
S211、根据所述第一图像帧对应的第一旋转矩阵、所述第二图像帧对应的第二旋转矩阵及对极几何原理，计算目标旋转矩阵。
其中，对极几何即为立体成像的基本几何。目标旋转矩阵可以用于对第一图像帧和第二图像帧进行旋转直至共面，且与基线平行。其中，基线可以根据第二图像帧与第一图像帧之间图像位置差的绝对值确定，例如，在一种可选的实现方式中，所述基线可以为第二图像帧与第一图像帧之间图像位置差的绝对值。
S212、根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵。
其中，第一新相机投影矩阵可以是根据目标旋转矩阵重新计算获得的，与第一图像帧对应的相机投影矩阵，第二新相机投影矩阵可以是根据目标旋转矩阵重新计算获得的，与第二图像帧对应的相机投影矩阵。
在一个可选的实现方式中，所述根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵，可以包括：基于拍摄所述第一图像帧的相机的相机内参、所述目标旋转矩阵和所述第一图像帧的图像位置，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；基于拍摄所述第二图像帧的相机的相机内参、所述目标旋转矩阵和所述第二图像帧的图像位置，计算与所述第二图像帧对应的第二新相机投影矩阵P n2。
例如，可以根据公式：P n1＝K[R n|-R nC 1]，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；根据公式：P n2＝K[R n|-R nC 2]，计算与所述第一图像帧对应的第一新相机投影矩阵P n2。其中，K为相机内参，R n为所述目标旋转矩阵，C 1为所述第一图像帧的图像位置，C 2为所述第二图像帧的图像位置，
S213、根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵。
其中，第一矫正变换矩阵用于对第一图像帧进行矫正变换，第二矫正变换矩阵用于对第二图像帧进行矫正变换。图片矫正是空间直角坐标系的变换，从一个二维坐标变换到另一个二维坐标，图片矫正是一个线性变换，保持了图像的“平行性”和“平直性”，即图像中原来的直线和平行线，变换后仍然保持原来的直线和平行线，映射变换比较常用的特殊变换有平移、缩放、翻转、旋转和剪切。矫正变换处理是指利用透视中心、像点、目标点三点共线的条件，按透视旋转定律使承影面(透视面)绕迹线(透视轴)旋转某一角度，破坏原有的投影光线束，仍能保持承影面上投影几何图形不变的变换。
在一个可选的实现方式中，所述根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵，可以包括：基于拍摄所述第一图像帧的相机的相机内参、所述第一新相机投影矩阵P n1和所述第一图像帧的旋转矩阵，计算第一矫正变换矩阵；基于拍摄所述第二图像帧的相机的相机内参、所述第二新相机投影矩阵P n2和所述第二图像帧的旋转矩阵，计算第二矫正变换矩阵。
例如，可以根据公式：T 1＝(P n1*(K*R' 1)')'，计算所述第一矫正变换矩阵T 1；根据公式：T 2＝(P n2*(K*R' 2)')'，计算所述第二矫正变换矩阵T 2。其中，K为相机内参，R 1为所述第一图像帧的旋转矩阵，R 2为所述第二图像帧的旋转矩阵。
S214、根据所述第一矫正变换矩阵对所述第一图像帧进行重采样映射变换，得到第一变换图像帧。
S215、根据所述第二矫正变换矩阵对所述第二图像帧进行重采样映射变换，得到第二变换图像帧。
其中，重采样映射变换即为根据矫正变换矩阵对图像帧中的像素点坐标按照一定规则进行重采样，并根据重采样得到的像素点坐标通过预设函数关系进行映射变换。
由于针对第一图像帧和第二图像帧确定了新的目标旋转矩阵，因此第一图像帧和第二图像帧原有的相机投影矩阵以及图像中各个像素点的像素位置都相应发生变化，需要针对第一图像帧和第二图像帧重新获取新的图像帧。相应的，在根据新相机投影矩阵得到第一矫正变换矩阵和第二矫正变换矩阵后，需要根据第一矫正变换矩阵对第一图像帧进行重采样映射变换，得到第一变换图像帧。同时，根据第二矫正变换矩阵对第二图像帧进行重采样映射变换，得到第二变换图像帧。得到的第一变换图像帧和第二变换图像帧满足双目立体匹配的条件。
S220、根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图。
在本申请实施例中，根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，可以是：利用相关匹配算法计算第一变换图像帧对应的视差图。其中，相关匹配算法包括但不限于SGM(Semi-Global Matching，半全局匹配)算法、局部立体匹配算法及全局立体匹配算法等。
在一个可选的实现方式中，所述根据所述视差图，计算与所述第一变换图像帧对应的深度图，可以包括：根据公式：
Figure PCTCN2021077689-appb-000002
计算与所述第一变换图像帧对应的深度图depth。其中，f为相机内参的焦距值，disparity为视差值，baseline为基线，baseline＝|C 2-C 1|。
在一个可选的实现方式中，在所述根据所述视差图，计算与所述第一变换图像帧对应的深度图之后，还可以包括：将所述第一变换图像帧对应的深度图与相邻预设数量的深度图进行一致性检测；根据一致性检测结果删除所述第一变换图像帧对应的深度图中的误匹配点数据。
其中，预设数量可以根据实际需求设定，示例性的，预设数量可以是10、12或15等，本申请实施例并不对预设数量的具体数值进行限定。
可以理解的是，由于相关匹配算法的准确率等多种因素的影响，第一变换图像帧对应的深度图中不可避免的会存在一些错误点数据。而针对短时间内生成的多个深度图，其对应的地面区域通常具有较大的相似性，其深度数据也应当具有一定的相似性。因此，为了进一步保证深度图数据的准确性和可靠性，可以将第一变换图像帧对应的深度图与相邻预设数量的深度图进行一致性检测；根据一致性检测结果删除第一变换图像帧对应的深度图中的误匹配点数据。
需要说明的是，如果第一变换图像帧对应的第一图像帧为首帧图像，没有相邻的深度图。或者，第一变换图像帧相邻的深度图数量较少，无法满足一致性检测的需求，如要求预设数量为12，而第一变换图像帧相邻的深度图数量为8。在上述情况下，可以不对第一变换图像帧对应的深度图进行一致性检测。
S230、根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
在一个可选的实现方式中，所述根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据，可以包括：根据下述公式：Point3d(X,Y,Z)＝P' n1*(x,y,depth)，计算所述第一图像帧对应的点云数据Point3d(X,Y,Z)；其中，x、y为所述第一变换图像帧中各像素点的水平坐标。
采用上述技术方案，通过根据第一图像帧对应的第一旋转矩阵、第二图像帧对应的第二旋转矩阵及对极几何原理，计算目标旋转矩阵，根据目标旋转矩阵，计算与第一图像帧对应的第一新相机投影矩阵，以及第二图像帧对应的第二新相机投影矩阵，根据第一新相机投影矩阵、第二新相机投影矩阵分别计算第一矫正变换矩阵及第二矫正变换矩阵，再根据第一矫正变换矩阵、第二矫正变换矩阵分别对第一图像帧、第二图像帧进行重采样映射变换，得到第一变换图像帧和第二变换图像帧，以根据第一变换图像帧和第二变换图像帧计算第一变换图像帧对应的点云数据，可以实现实时动态地生成整个场景的三维点云。并且在计算得到第一变换图像帧对应的深度图后，通过一致性检测删除深度图中的误匹配点，从而保证深度图数据的准确性，进而保证点云数据的准确性。
需要说明的是，以上各实施例中各技术特征之间的任意排列组合也属于本申请的保护范围。
图3是本申请实施例提供的一种点云数据的实时显示方法的流程图，该方法可适用于实时显示点云数据的情况，该方法可以由点云数据的实时显示装置来执行，该装置可以由软件和/或硬件的方式来实现，并一般可集成在计算机设备中。相应的，如图3所示，该方法包括如下操作：
S310、获取无人机实时拍摄得到的图像帧。
无人机在飞行作业的过程中，可以利用摄像装置实时拍照获取多个图像帧。
S320、按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧，其中，所述第一图像帧初始化为无人机采集的首帧图像。
在本申请实施例中，为了对整个场景的点云数据进行实时生成和显示，可以按照图像帧的拍摄顺序，从首帧图像开始获取第一图像帧，并获取与第一图像帧相邻的第二图像帧。
在一个可选的实现方式中，点云数据的实时显示方法还可以包括：如果确定当前处理的第一图像帧与第二图像帧未处于同一直线上，则可以将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作。其中，当前处理的第一图像帧与第二图像帧未处于同一直线上可以理解为当前处理的第一图像帧和第二图像帧的几何相关性较弱，很难被调整为符合双目立体匹配条件的图像。
在本申请实施例中，如果当前处理的第一图像帧和第二图像帧处于同一直线上，则可以继续处理。相应的，如果确定当前处理的第一图像帧与第二图像帧未处于同一直线上，则将第二图像帧更新为第一图像帧后，获取与第二图像帧相邻的第三图像帧更新为第二图像帧并继续处理。
S330、按照上述任意实施例所提供的点云数据的生成方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示。
确定第一图像帧和第二图像帧后，即可按照上述任意实施例所提供的点云数据的生成方法所示计算得到与第一图像帧对应的点云数据，然后对实时计算得到的点云数据进行实时显示。
S340、将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
相应的，当第一图像帧生成对应的点云数据后，可以将第二图像帧更新为第一图像帧，并将与第二图像帧相邻的第三图像帧更新为第二图像帧继续处理，直到完成对全部图像帧的处理。由此可见，本申请实施例可以从首帧图像开始处理，实时生成每帧图像的点云数据并进行显示。可以理解的是，每当一帧图像实时生成点云数据并显示后，整个场景就会实时增加该帧图像的点云数据，因此，本申请实施例所提供的点云数据的实时显示方法可以实现对整个场景点云数据的增量式显示。
本申请实施例通过在实时生成场景的点云数据后，对点云数据进行实时显示，能够解决现有点云数据生成方法存在的无法实时显示生成整个场景的三维点云的问题，从而实时增量式显示整个场景的三维点云。
需要说明的是，在其他实施方式中，上述点云数据的实时显示方法可以融合到上述点云数据的生成方法中，可以理解为：上述点云数据的实时显示方法可以作为上述点云数据的生成方法所包含的步骤，例如，在计算得到点云数据之后，本申请上述任意实施例所提供的点云数据的生成方法还可以包括：对所述点云数据进行实时显示。
其中，将点云数据的实时显示方法融合到点云数据的生成方法之后，点云数据的实时显示方法所包含的步骤可以进行适应性调整，例如，可以将步骤S310、S320删除，将步骤S330适应性地调整为：对所述点云数据进行实时显示，可以理解为：在点云数据的生成方法中，由于已经通过步骤S110、S120和S130进行点云数据的计算，所以为显示已经计算得到的点云数据，只要在计算得到点云数据之后，对所述点云数据进行实时显示即可。
图4是本申请实施例提供的一种点云数据的生成装置的示意图，如图4所示，所述装置包括：图像帧获取模块410、深度图计算模块420以及点云数据计算模块430，其中：
图像帧获取模块410，用于获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；
深度图计算模块420，用于根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；
点云数据计算模块430，用于根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
本申请实施例通过对所拍摄的第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，并根据第一变换图像帧和第二变换图像帧计算与第一变换图像帧对应的视差图，根据视差图计算与第一变换图像帧对应的深度图，以根据深度图以及第一变换图像帧计算与第一图像帧对应的点云数据，解决现有点云数据生成方法存在的无法实时动态地生成整个场景的三维点云的问题，从而实时动态地生成整个场景的三维点云。
可选的，图像帧获取模块410，包括第一图像帧获取单元，用于如果确定所述第一图像帧与所述第二图像帧处于同一直线上，则将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。
可选的，所述第一图像帧和第二图像帧为无人机沿设定航线飞行时所拍摄得到的图像帧；第一图像帧获取单元，具体用于获取与所述第一图像帧对应的第一旋转矩阵，以及与所述第二图像帧对应的第二旋转矩阵；计算与所述第一旋转矩阵对应的第一偏航角，以及与所述第二旋转矩阵对应的第二偏航角；如果所述第一偏航角与所述第二偏航角之间的差异值在预设的差异值范围内，则确定所述第一图像帧与所述第二图像帧处于同一直线。
可选的，图像帧获取模块410，还可以包括：
目标旋转矩阵计算单元，用于根据所述第一图像帧对应的第一旋转矩阵、所述第二图像帧对应的第二旋转矩阵及对极几何原理，计算目标旋转矩阵；
新相机投影矩阵计算单元，用于根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵；
矫正变换矩阵计算单元，用于根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵；
第一变换图像帧获取单元，用于根据所述第一矫正变换矩阵对所述第一图像帧进行重采样映射变换，得到第一变换图像帧；
第二变换图像帧获取单元，用于根据所述第二矫正变换矩阵对所述第二图像帧进行重采样映射变换，得到第二变换图像帧。
可选的，新相机投影矩阵计算单元，具体用于基于拍摄所述第一图像帧的相机的相机内参、所述目标旋转矩阵和所述第一图像帧的图像位置，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；基于拍摄所述第二图像帧的相机的相机内参、所述目标旋转矩阵和所述第二图像帧的图像位置，计算与所述第二图像帧对应的第二新相机投影矩阵P n2。
例如，根据公式：P n1＝K[R n|-R nC 1]，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；根据公式：P n2＝K[R n|-R nC 2]，计算与所述第一图像帧对应的第一新相机投影矩阵P n2。
矫正变换矩阵计算单元，具体用于基于拍摄所述第一图像帧的相机的相机内参、所述第一新相机投影矩阵P n1和所述第一图像帧的旋转矩阵，计算第一矫正变换矩阵；基于拍摄所述第二图像帧的相机的相机内参、所述第二新相机投影矩阵P n2和所述第二图像帧的旋转矩阵，计算第二矫正变换矩阵。
例如，根据公式：T 1＝(P n1*(K*R' 1)')'，计算与所述第一矫正变换矩阵T 1；根据公式：T 2＝(P n2*(K*R' 2)')'，计算与所述第二矫正变换矩阵T 2。
深度图计算模块，具体用于根据公式：
Figure PCTCN2021077689-appb-000003
计算与所述第一变换图像帧对应的深度图depth；点云数据计算模块，具体用于根据公式：Point3d(X,Y,Z)＝P' n1*(x,y,depth)，计算所述第一图像帧对应的点云数据Point3d(X,Y,Z)；其中，K为相机内参，R n为所述目标旋转矩阵，C 1为所述第一图像帧的图像位置，C 2为所述第二图像帧的图像位置，R 1为所述第一图像帧的旋转矩阵，R 2为所述第二图像帧的旋转矩阵，f为相机内参的焦距值，disparity为视差值，baseline为基线，baseline＝|C 2-C 1|，x、y为所述第一变换图像帧中各像素点的水平坐标。
可选的，点云数据的生成装置还包括：一致性检测模块，用于将所述第一变换图像帧对应的深度图与相邻预设数量的深度图进行一致性检测；误匹配点数据删除模块，用于根据一致性检测结果删除所述第一变换图像帧对应的深度图中的误匹配点数据。
可选的，图像帧获取模块410，还可以包括第二图像帧获取单元，用于如果确定所述第一图像帧与所述第二图像帧未处于同一直线上，则采用所述第二图像帧更新所述第一图像帧；返回执行获取拍摄的第一图像帧和第二图像帧的操作，直至完成对全部图像帧的处理。
上述点云数据的生成装置可执行本申请任意实施例所提供的点云数据的生成方法，具备执行方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节，可参见本申请任意实施例提供的点云数据的生成方法。
由于上述所介绍的点云数据的生成装置为可以执行本申请实施例中的点云数据的生成方法的装置，故而基于本申请实施例中所介绍的点云数据的生成方法，本领域所属技术人员能够了解本实施例的点云数据的生成装置的具体实施方式以及其各种变化形式，所以在此对于该点云数据的生成装置如何实现本申请实施例中的点云数据的生成方法不再详细介绍。只要本领域所属技术人员实施本申请实施例中点云数据的生成方法所采用的装置，都属于本申请所欲保护的范围。
图5是本申请实施例提供的一种点云数据的实时显示装置的示意图，如图5所示，所述装置包括：图像帧获取模块510、相邻图像帧确定模块520、点云数据显示模块530以及返回执行模块540，其中：
图像帧获取模块510，用于获取无人机实时拍摄得到的图像帧；
相邻图像帧确定模块520，用于按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧；
点云数据显示模块530，用于按照上述任意实施例所提供的点云数据的生成方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；
返回执行模块540，用于将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
可选的，点云数据的实时显示装置还可以包括：图像帧更新模块，用于如果确定当前处理的第一图像帧与第二图像帧未处于同一直线上，则将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作。
本申请实施例通过在实时生成场景的点云数据后，对点云数据进行实时显示，能够解决现有点云数据生成方法存在的无法实时显示生成整个场景的三维点云的问题，从而实时增量式显示整个场景的三维点云。
上述点云数据的实时显示装置可执行本申请任意实施例所提供的点云数据的实时显示方法，具备执行方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节，可参见本申请任意实施例提供的点云数据的实时显示方法。
由于上述所介绍的点云数据的实时显示装置为可以执行本申请实施例中的点云数据的实时显示方法的装置，故而基于本申请实施例中所介绍的点云数据的实时显示方法，本领域所属技术人员能够了解本实施例的点云数据的实时显示装置的具体实施方式以及其各种变化形式，所以在此对于该点云数据的实时显示装置如何实现本申请实施例中的点云数据的实时显示方法不再详细介绍。只要本领域所属技术人员实施本申请实施例中点云数据的实时显示方法所采用的装置，都属于本申请所欲保护的范围。
需要说明的是，在其他实施方式中，上述点云数据的实时显示装置可以融合到上述点云数据的生成装置中，可以理解为：上述点云数据的实时显示装置可以作为上述点云数据的生成装置所包含的模块，例如，本申请上述任意实施例所提供的点云数据的生成装置还可以包括：所述点云数据显示模块和所述返回执行模块。
其中，将点云数据的实时显示装置融合到点云数据的生成装置之后，点云数据的实时显示装置所包含的模块的功能可以进行适应性调整，例如，点云数据显示模块可以直接显示由点云数据计算模块计算所得的点云数据，可以理解为：点云数据显示模块可以不用再进行点云数据的计算，只负责显示点云数据。
图6为本申请实施例提供的一种计算机设备的结构示意图。图6示出了适于用来实现本申请实施方式的计算机设备612的框图。图6显示的计算机设备612仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图6所示，计算机设备612以通用计算设备的形式表现。计算机设备612的组件可以包括但不限于：一个或者多个处理器616，存储装置628，连接不同系统组件(包括存储装置628和处理器616)的总线618。
总线618表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(Industry Standard Architecture，ISA)总线，微通道体系结构(MicroChannel Architecture，MCA)总线，增强型ISA总线、视频电子标准协会(Video Electronics Standards Association，VESA)局域总线以及外围组件互连(Peripheral Component Interconnect，PCI)总线。
计算机设备612典型地包括多种计算机系统可读介质。这些介质可以是任何能够被计算机设备612访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。
存储装置628可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(Random Access Memory，RAM)630和/或高速缓存存储器632。计算机设备612可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机系统存储介质。仅作为举例，存储系统634可以用于读写不可移动的、非易失性磁介质(图6未显示，通常称为“硬盘驱动器”)。尽管图6中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如只读光盘(Compact Disc-ReadOnly Memory，CD-ROM)、数字视盘(Digital Video Disc-Read Only Memory，DVD-ROM)或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与总线618相连。存储装置628可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本申请各实施例的功能。
具有一组(至少一个)程序模块626的程序636，可以存储在例如存储装置628中，这样的程序模块626包括但不限于操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块626通常执行本申请所描述的实施例中的功能和/或方法。
计算机设备612也可以与一个或多个外部设备614(例如键盘、指向设备、摄像头、显示器624等)通信，还可与一个或者多个使得用户能与该计算机设备612交互的设备通信，和/或与使得该计算机设备612能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(Input/Output，I/O)接口622进行。并且，计算机设备612还可以通过网络适配器620与一个或者多个网络(例如局域网(Local Area Network，LAN)，广域网Wide Area Network，WAN)和/或公共网络，例如因特网)通信。如图6所示，网络适配器620通过总线618与计算机设备612的其它模块通信。应当明白，尽管图中未示出，可以结合计算机设备612使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、磁盘阵列(Redundant Arrays of Independent Disks，RAID)系统、磁带驱动器以及数据备份存储系统等。
处理器616通过运行存储在存储装置628中的程序，从而执行各种功能应用以及数据处理，例如实现本申请上述实施例所提供的点云数据的生成方法。
也即，所述处理单元执行所述程序时实现：获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
本申请实施例还提供了一种用于执行本申请实施所提供的点云数据的实时显示方法的计算机设备，该设备包括：一个或多个处理器；存储装置，用于存储一个或多个程序；当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本申请实施例所提供的点云数据的实时显示方法：获取无人机实时拍摄得到的图像帧；按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧，其中，所述第一图像帧初始化为无人机采集的首帧图像；按照本申请任意实施例所提供的点云数据的生成方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。该计算机设备具体结构以及细节内容可参照图6所示的计算机设备。
本申请实施例还提供一种存储计算机程序的计算机存储介质，所述计算机程序在由计算机处理器执行时用于执行本申请上述任一实施例所述的点云数据的生成方法：获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立 体匹配条件的第一变换图像帧和第二变换图像帧；根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
本申请实施例的计算机存储介质，可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(RAM)、只读存储器(Read Only Memory，ROM)、可擦式可编程只读存储器((Erasable Programmable Read Only Memory，EPROM)或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于无线、电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言或其组合来编写用于执行本申请操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言——诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时可实现上述实施例所提供的点云数据的实时显示方法：获取无人机实时拍摄得到的图像帧；按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧，其中，所述第一图像帧初始化为无人机采集的首帧图像；按照本申请上述任意实施例所提供的点云数据的生成方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
注意，上述仅为本申请的可选实施例及所运用技术原理。本领域技术人员会理解，本申请不限于这里所述的特定实施例，对本领域技术人员来说能够进行各种明显的变化、重新调整和替代的技术方案均属于本申请的保护范围。因此，虽然通过以上实施例对本申请进行了较为详细的说明，但是本申请不仅仅限于以上实施例，在不脱离本申请构思的情况下，还可以包括更多其他等效实施例，而本申请的范围由所附的权利要求范围决定。
工业实用性
本申请实施例通过对拍摄的第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，并根据第一变换图像帧和第二变换图像帧计算与 第一变换图像帧对应的视差图，根据视差图计算与第一变换图像帧对应的深度图，以根据深度图以及第一变换图像帧计算与第一图像帧对应的点云数据，并实时显示该点云数据，解决现有点云数据处理方法存在的无法实时动态地生成并显示整个场景的三维点云的问题，从而实时动态地生成并显示整个场景的三维点云。

Claims (21)

  1. 一种点云数据的生成方法，其特征在于，包括：
    获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；
    根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；
    根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
  2. 根据权利要求1所述的方法，其特征在于，获取相邻拍摄的两个图像帧作为所述第一图像帧和第二图像帧。
  3. 根据权利要求1或2所述的方法，其特征在于，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
    如果确定所述第一图像帧与所述第二图像帧处于同一直线上，则将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧。
  4. 根据权利要求3所述的方法，其特征在于，所述第一图像帧和第二图像帧为无人机沿设定航线飞行时所拍摄得到的图像帧；
    确定所述第一图像帧与所述第二图像帧处于同一直线的过程包括：
    获取与所述第一图像帧对应的第一旋转矩阵，以及与所述第二图像帧对应的第二旋转矩阵；
    计算与所述第一旋转矩阵对应的第一偏航角，以及与所述第二旋转矩阵对应的第二偏航角；
    如果所述第一偏航角与所述第二偏航角之间的差异值在预设的差异值范围内，则确定所述第一图像帧与所述第二图像帧处于同一直线。
  5. 根据权利要求1-4任一项所述的方法，其特征在于，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
    根据所述第一图像帧对应的第一旋转矩阵、所述第二图像帧对应的第二旋转矩阵及对极几何原理，计算目标旋转矩阵；
    根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵；
    根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵；
    根据所述第一矫正变换矩阵对所述第一图像帧进行重采样映射变换，得到第一变换图像帧；
    根据所述第二矫正变换矩阵对所述第二图像帧进行重采样映射变换，得到第二变换图像帧。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述目标旋转矩阵，计算与所述第一图像帧对应的第一新相机投影矩阵，以及所述第二图像帧对应的第二新相机投影矩阵，包括：
    基于拍摄所述第一图像帧的相机的相机内参、所述目标旋转矩阵和所述第一图像帧的图像位置，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；
    基于拍摄所述第二图像帧的相机的相机内参、所述目标旋转矩阵和所述第二图像帧的图像位置，计算与所述第二图像帧对应的第二新相机投影矩阵P n2。
  7. 根据权利要求6所述的方法，其特征在于，所述基于拍摄所述第一图像帧的相机的相机内参、所述目标旋转矩阵和所述第一图像帧的图像位置，计算与所述第一图像帧对应的第一新相机投影矩阵P n1，包括：
    根据公式：P n1＝K[R n|-R nC 1]，计算与所述第一图像帧对应的第一新相机投影矩阵P n1；
    所述基于拍摄所述第二图像帧的相机的相机内参、所述目标旋转矩阵和所述第二图像帧的图像位置，计算与所述第二图像帧对应的第二新相机投影矩阵P n2，包括：
    根据公式：P n2＝K[R n|-R nC 2]，计算与所述第二图像帧对应的第二新相机投影矩阵P n2；
    其中，K为相机内参，R n为所述目标旋转矩阵，C 1为所述第一图像帧的图像位置，C 2为所述第二图像帧的图像位置。
  8. 根据权利要求5-7任一项所述的方法，其特征在于，
    所述根据所述第一新相机投影矩阵计算第一矫正变换矩阵，根据所述第二新相机投影矩阵计算第二矫正变换矩阵，包括：
    基于拍摄所述第一图像帧的相机的相机内参、所述第一新相机投影矩阵P n1和所述第一图像帧的旋转矩阵，计算第一矫正变换矩阵；
    基于拍摄所述第二图像帧的相机的相机内参、所述第二新相机投影矩阵P n2和所述第二图像帧的旋转矩阵，计算第二矫正变换矩阵。
  9. 根据权利要求8所述的方法，其特征在于，
    所述基于拍摄所述第一图像帧的相机的相机内参、所述第一新相机投影矩阵P n1和所述第一图像帧的旋转矩阵，计算第一矫正变换矩阵，包括：
    根据公式：T 1＝(P n1*(K*R' 1)')'，计算所述第一矫正变换矩阵T 1；
    所述基于拍摄所述第二图像帧的相机的相机内参、所述第二新相机投影矩阵P n2和所述第二图像帧的旋转矩阵，计算第二矫正变换矩阵，包括：
    根据公式：T 2＝(P n2*(K*R' 2)')'，计算所述第二矫正变换矩阵T 2；
    其中，K为相机内参，R 1为所述第一图像帧的旋转矩阵，R 2为所述第二图像帧的旋转矩阵。
  10. 根据权利要求1-9任一项所述的方法，其特征在于，
    所述根据所述视差图，计算与所述第一变换图像帧对应的深度图，包括：
    根据公式：
    Figure PCTCN2021077689-appb-100001
    计算与所述第一变换图像帧对应的深度图depth；
    所述根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据，包括：
    根据公式：Point3d(X,Y,Z)＝P' n1*(x,y,depth)，计算所述第一图像帧对应的点云数据Point3d(X,Y,Z)；
    其中，f为相机内参的焦距值，disparity为视差值，baseline为基线，baseline＝|C 2-C 1|，x、y为所述第一变换图像帧中各像素点的水平坐标，C 1为所述第一图像帧的图像位置，C 2为所述第二图像帧的图像位置。
  11. 根据权利要求1-8任意一项所述的方法，其特征在于，在所述根据所述视差图，计算与所述第一变换图像帧对应的深度图之后，还包括：
    将所述第一变换图像帧对应的深度图与相邻预设数量的深度图进行一致性检测；
    根据一致性检测结果删除所述第一变换图像帧对应的深度图中的误匹配点数据。
  12. 根据权利要求3所述的方法，其特征在于，将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
    如果确定所述第一图像帧与所述第二图像帧未处于同一直线上，则采用所述第二图像帧更新所述第一图像帧；
    返回执行获取拍摄的第一图像帧和第二图像帧的操作。
  13. 根据权利要求3所述的方法，其特征在于，所述方法，包括：
    如果确定所述第一图像帧与所述第二图像帧未处于同一直线上，则将所述第二图像帧作为新的第一图像帧，并于拍摄时间在所述第二图像帧之后的图像帧中确定新的第二图像帧，以根据所述新的第一图像帧和新的第二图像帧计算与所述新的第一图像帧对应的点云数据。
  14. 根据权利要求1-4任一项所述的方法，其特征在于，所述第一图像帧和第二图像帧为无人机飞行时所拍摄得到的图像帧；将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧，包括：
    获取拍摄所述第一图像帧时所述无人机的第一旋转矩阵，以及拍摄所述第二图像帧时所述无人机的第二旋转矩阵；
    根据所述第一旋转矩阵及所述第二旋转矩阵，对所述第一图像帧和所述第二图像帧进行变换处理，使由所述第一图像帧和第二图像帧中共轭像素点确定的成对共轭线变为共线且与所述第一图像帧和所述第二图像帧之间的基线平行。
  15. 一种点云数据的实时显示方法，其特征在于，包括：
    获取无人机实时拍摄得到的图像帧；
    按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧；
    按照如权利要求1-14任一项所述的方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；
    将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
  16. 根据权利要求15所述的方法，其特征在于，还包括：
    如果确定当前处理的第一图像帧与第二图像帧未处于同一直线上，则将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作。
  17. 根据权利要求15所述的方法，其特征在于，在获取所述第一图像帧后，所述方法还包括：
    将所述第一图像帧初始化为无人机采集的首帧图像。
  18. 一种点云数据的生成装置，其特征在于，包括：
    图像帧获取模块，用于获取拍摄的第一图像帧和第二图像帧，并将所述第一图像帧和第二图像帧调整为满足双目立体匹配条件的第一变换图像帧和第二变换图像帧；
    深度图计算模块，用于根据所述第一变换图像帧和所述第二变换图像帧，计算与第一变换图像帧对应的视差图，并根据所述视差图，计算与所述第一变换图像帧对应的深度图；
    点云数据计算模块，用于根据所述深度图以及所述第一变换图像帧，计算与所述第一图像帧对应的点云数据。
  19. 一种点云数据的实时显示装置，其特征在于，包括：
    图像帧获取模块，用于获取无人机实时拍摄得到的图像帧；
    相邻图像帧确定模块，用于按照图像帧的拍摄顺序，获取第一图像帧，并获取与第一图像帧相邻的第二图像帧；
    点云数据显示模块，用于按照如权利要求1-14任一项所述的方法计算得到与所述第一图像帧对应的点云数据，并对所述点云数据进行实时显示；
    返回执行模块，用于将所述第二图像帧作为新的第一图像帧后，返回执行获取与第一图像帧相邻的第二图像帧的操作，直至完全对所拍摄的全部图像帧的处理。
  20. 一种计算机设备，其特征在于，所述计算机设备包括：
    一个或多个处理器；
    存储装置，用于存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-14中任一所述的点云数据的生成方法，或实现如权利要求15-17中任一所述的点云数据的实时显示方法。
  21. 一种计算机存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现如权利要求1-14中任一所述的点云数据的生成方法，或实现如权利要求15-17中任一所述的点云数据的实时显示方法。
PCT/CN2021/077689 2020-03-18 2021-02-24 点云数据的生成及实时显示方法、装置、设备及介质 WO2021185036A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010191155.3A CN113496503B (zh) 2020-03-18 2020-03-18 点云数据的生成及实时显示方法、装置、设备及介质
CN202010191155.3 2020-03-18

Publications (1)

Publication Number Publication Date
WO2021185036A1 true WO2021185036A1 (zh) 2021-09-23

Family

ID=77769175

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/077689 WO2021185036A1 (zh) 2020-03-18 2021-02-24 点云数据的生成及实时显示方法、装置、设备及介质

Country Status (2)

Country Link
CN (1) CN113496503B (zh)
WO (1) WO2021185036A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115223028A (zh) * 2022-06-02 2022-10-21 支付宝(杭州)信息技术有限公司 场景重建及模型训练方法、装置、设备、介质及程序产品
CN116906277A (zh) * 2023-06-20 2023-10-20 北京图知天下科技有限责任公司 风机偏航变动的确定方法、装置、电子设备及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114187357A (zh) * 2021-12-10 2022-03-15 北京百度网讯科技有限公司 一种高精地图的生产方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180018805A1 (en) * 2016-07-13 2018-01-18 Intel Corporation Three dimensional scene reconstruction based on contextual analysis
CN109741385A (zh) * 2018-12-24 2019-05-10 浙江大华技术股份有限公司 一种图像处理系统、方法、装置、电子设备及存储介质
CN110555878A (zh) * 2018-05-31 2019-12-10 上海微电子装备（集团）股份有限公司 物体空间位置形态的确定方法、装置、存储介质及机器人
CN110800023A (zh) * 2018-07-24 2020-02-14 深圳市大疆创新科技有限公司 图像处理方法和设备、摄像装置以及无人机
CN110827770A (zh) * 2019-12-24 2020-02-21 惠州视维新技术有限公司 显示图像的区域背光亮度调节方法、装置和存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101557534B (zh) * 2009-05-19 2010-09-01 无锡景象数字技术有限公司 一种从视频相近帧中生成视差图的方法
CN101577004B (zh) * 2009-06-25 2013-05-15 海信集团有限公司 一种极线矫正方法、装置和系统
CN101625768B (zh) * 2009-07-23 2011-11-09 东南大学 一种基于立体视觉的三维人脸重建方法
US10706569B2 (en) * 2016-06-08 2020-07-07 Amazon Technologies, Inc. Selectively paired imaging elements for stereo images
WO2019182974A2 (en) * 2018-03-21 2019-09-26 Nvidia Corporation Stereo depth estimation using deep neural networks
CN108765481B (zh) * 2018-05-25 2021-06-11 亮风台（上海）信息科技有限公司 一种单目视频的深度估计方法、装置、终端和存储介质
CN110728710B (zh) * 2018-07-16 2023-10-27 株式会社理光 视觉里程计算方法、装置和计算机可读存储介质
CN109472830A (zh) * 2018-09-28 2019-03-15 中山大学 一种基于无监督学习的单目视觉定位方法
CN110223354A (zh) * 2019-04-30 2019-09-10 惠州市德赛西威汽车电子股份有限公司 一种基于sfm三维重建的相机自标定方法
CN110375732A (zh) * 2019-07-22 2019-10-25 中国人民解放军国防科技大学 基于惯性测量单元和点线特征的单目相机位姿测量方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180018805A1 (en) * 2016-07-13 2018-01-18 Intel Corporation Three dimensional scene reconstruction based on contextual analysis
CN110555878A (zh) * 2018-05-31 2019-12-10 上海微电子装备（集团）股份有限公司 物体空间位置形态的确定方法、装置、存储介质及机器人
CN110800023A (zh) * 2018-07-24 2020-02-14 深圳市大疆创新科技有限公司 图像处理方法和设备、摄像装置以及无人机
CN109741385A (zh) * 2018-12-24 2019-05-10 浙江大华技术股份有限公司 一种图像处理系统、方法、装置、电子设备及存储介质
CN110827770A (zh) * 2019-12-24 2020-02-21 惠州视维新技术有限公司 显示图像的区域背光亮度调节方法、装置和存储介质

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115223028A (zh) * 2022-06-02 2022-10-21 支付宝(杭州)信息技术有限公司 场景重建及模型训练方法、装置、设备、介质及程序产品
CN115223028B (zh) * 2022-06-02 2024-03-29 支付宝(杭州)信息技术有限公司 场景重建及模型训练方法、装置、设备、介质及程序产品
CN116906277A (zh) * 2023-06-20 2023-10-20 北京图知天下科技有限责任公司 风机偏航变动的确定方法、装置、电子设备及存储介质
CN116906277B (zh) * 2023-06-20 2024-07-30 北京图知天下科技有限责任公司 风机偏航变动的确定方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN113496503B (zh) 2022-11-08
CN113496503A (zh) 2021-10-12

Similar Documents

Publication Publication Date Title
WO2021185036A1 (zh) 点云数据的生成及实时显示方法、装置、设备及介质
WO2019205852A1 (zh) 确定图像捕捉设备的位姿的方法、装置及其存储介质
WO2021227359A1 (zh) 一种无人机投影方法、装置、设备及存储介质
WO2019161813A1 (zh) 动态场景的三维重建方法以及装置和系统、服务器、介质
US20210141378A1 (en) Imaging method and device, and unmanned aerial vehicle
WO2019242262A1 (zh) 基于增强现实的远程指导方法、装置、终端和存储介质
EP2328125B1 (en) Image splicing method and device
KR100793838B1 (ko) 카메라 모션 추출장치, 이를 이용한 해상장면의 증강현실 제공 시스템 및 방법
CN106846467B (zh) 基于每个相机位置优化的实体场景建模方法和系统
US11935187B2 (en) Single-pass object scanning
CN112927362B (zh) 地图重建方法及装置、计算机可读介质和电子设备
WO2019164498A1 (en) Methods, devices and computer program products for global bundle adjustment of 3d images
CN109191554B (zh) 一种超分辨图像重建方法、装置、终端和存储介质
US11315313B2 (en) Methods, devices and computer program products for generating 3D models
CN111127524A (zh) 一种轨迹跟踪与三维重建方法、系统及装置
KR102200866B1 (ko) 2차원 이미지를 이용한 3차원 모델링 방법
JP7082713B2 (ja) ローリングシャッター画像／ビデオのｓｆｍ／ｓｌａｍに対するアプリケーションで畳み込みニューラルネットワークを用いる画像／ビデオにおけるローリングシャッター補正
GB2567245A (en) Methods and apparatuses for depth rectification processing
CN113129352A (zh) 一种稀疏光场重建方法及装置
WO2022041119A1 (zh) 三维点云处理方法及装置
WO2021051220A1 (zh) 一种点云融合方法、设备、系统及存储介质
CN113129422A (zh) 一种三维模型构建方法、装置、存储介质和计算机设备
US10721419B2 (en) Ortho-selfie distortion correction using multiple image sensors to synthesize a virtual image
CN114089836B (zh) 标注方法、终端、服务器和存储介质
KR102312471B1 (ko) 입체 이미지 캡처

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21772408

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 23.02.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21772408

Country of ref document: EP

Kind code of ref document: A1