WO2021227359A1 - 一种无人机投影方法、装置、设备及存储介质 - Google Patents

一种无人机投影方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2021227359A1
WO2021227359A1 PCT/CN2020/121663 CN2020121663W WO2021227359A1 WO 2021227359 A1 WO2021227359 A1 WO 2021227359A1 CN 2020121663 W CN2020121663 W CN 2020121663W WO 2021227359 A1 WO2021227359 A1 WO 2021227359A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
video frame
feature
camera
posture
Prior art date
Application number
PCT/CN2020/121663
Other languages
English (en)
French (fr)
Inventor
高星
徐建明
陈奇毅
石立阳
Original Assignee
佳都新太科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 佳都新太科技股份有限公司 filed Critical 佳都新太科技股份有限公司
Publication of WO2021227359A1 publication Critical patent/WO2021227359A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3141—Constructional details thereof

Definitions

  • the embodiments of the present application relate to the field of image processing, and in particular to a drone projection method, device, equipment, and storage medium.
  • Video projection technology combines surveillance video with a three-dimensional model, and projects the surveillance video of the area of interest into a three-dimensional model of a large scene, which can realize the combination of virtual and real static large scenes and dynamic key scenes.
  • the embodiments of the present application provide a drone projection method, device, equipment, and storage medium to meet the real-time requirements of video projection.
  • an embodiment of the present application provides a drone projection method, including:
  • the camera in the virtual scene is set according to the camera pose matrix, the focal length information and/or the distortion parameter, and the video frame is added to the rendering pipeline for video projection.
  • the initial position and posture of the drone camera in the world coordinate system is determined based on the positioning information, posture information, and pan/tilt information of the drone, and a three-dimensional map is rendered to obtain the initial position and posture corresponding to the initial position and posture.
  • Two-dimensional picture including:
  • the feature matching of the video frame captured by the drone with the two-dimensional picture includes:
  • Feature matching is performed on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
  • the method further includes:
  • the matching feature points are screened based on the RANSAC algorithm.
  • the method further includes:
  • the determining the three-dimensional feature points corresponding to the two-dimensional feature points on the video frame on the three-dimensional map according to the feature matching result includes:
  • the coordinates of the three-dimensional feature points corresponding to the two-dimensional feature points in the three-dimensional map are determined.
  • the determining the camera pose matrix, focal length information, and/or distortion parameters through a pose solving algorithm based on the two-dimensional feature points and the three-dimensional feature points includes:
  • the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera pose matrix, focal length information and/or distortion parameters.
  • an embodiment of the present application provides a drone projection device, including a two-dimensional rendering module, a feature corresponding module, a pose determination module, and a video projection module, wherein:
  • the two-dimensional rendering module is used to determine the initial position and posture of the UAV camera in the world coordinate system based on the positioning information, attitude information and pan-tilt information of the UAV, and to render the three-dimensional map to obtain the initial position and posture Corresponding two-dimensional picture;
  • the feature corresponding module is used to perform feature matching between the video frame taken by the drone and the two-dimensional picture, and determine the three-dimensional feature corresponding to the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result point;
  • the pose determination module is configured to determine the camera pose matrix, focal length information and/or distortion parameters through a pose solving algorithm based on the two-dimensional feature points and the three-dimensional feature points;
  • the video projection module is configured to set a camera in a virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
  • the two-dimensional rendering module is specifically used for:
  • the feature corresponding module when the feature corresponding module performs feature matching between the video frame captured by the drone and the two-dimensional picture, it specifically includes:
  • Feature matching is performed on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
  • the feature corresponding module after the feature corresponding module performs feature matching on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor, it also screens the matched feature points based on the RANSAC algorithm.
  • a time synchronization module which is used to obtain the initial position and posture and the time stamp of the video frame before the feature corresponding module performs feature matching between the video frame taken by the drone and the two-dimensional picture ; Perform time synchronization of the initial position and posture with the video frame according to the correspondence between the timestamps.
  • the feature corresponding module determines the corresponding three-dimensional feature point of the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result, it specifically includes:
  • the coordinates of the three-dimensional feature points corresponding to the two-dimensional feature points in the three-dimensional map are determined.
  • the pose determination module is specifically used for:
  • the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera pose matrix, focal length information and/or distortion parameters.
  • an embodiment of the present application provides a computer device, including: a memory and one or more processors;
  • the memory is used to store one or more programs
  • the one or more processors implement the drone projection method as described in the first aspect.
  • embodiments of the present application provide a storage medium containing computer-executable instructions, which are used to execute the drone projection method as described in the first aspect when the computer-executable instructions are executed by a computer processor .
  • This embodiment of the application first determines the initial position and posture of the drone camera based on the positioning information, attitude information, and pan/tilt information of the drone, and renders the three-dimensional map based on the initial position and posture to obtain the difference between the initial position and posture.
  • the two-dimensional image corresponding to the range captured by the human-machine camera, and then feature matching between the two-dimensional image and the video frame captured by the drone camera.
  • the three-dimensional feature corresponding to the two-dimensional feature point on the video frame is determined in the three-dimensional map Point, the camera pose matrix, focal length information and/or distortion parameters can be determined by the pose solving algorithm, the camera in the virtual scene is set according to the above information, and the video frame is added to the rendering pipeline for video projection, thereby realizing automatic real-time projection.
  • the video frame can be projected on the three-dimensional model by matching the video frame and the two-dimensional picture In the correct position, effectively improve the video projection effect.
  • FIG. 1 is a flowchart of a drone projection method provided by an embodiment of the present application
  • FIG. 2 is a flowchart of another drone projection method provided by an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of an unmanned aerial vehicle projection device provided by an embodiment of the present application.
  • Fig. 4 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
  • FIG. 1 shows a flowchart of a drone projection method provided by an embodiment of the application.
  • the drone projection method provided by the embodiment of the application can be executed by a drone projection device, which can Realized by means of hardware and/or software, and integrated in computer equipment.
  • the UAV projection method includes:
  • S101 Determine the initial position and posture of the UAV camera in the world coordinate system based on the UAV's positioning information, posture information and PTZ information, and render a three-dimensional map to obtain a two-dimensional picture corresponding to the initial position and posture .
  • the camera is mounted on the gimbal, and the gimbal is mounted on the drone.
  • the positioning information includes longitude, latitude and altitude, which can be detected by the GPS module mounted on the drone.
  • the attitude information includes pitch angle, roll angle and yaw angle, which can be detected by the IMU (Inertial Measurement Unit) mounted on the UAV.
  • the pan/tilt information (PTZ, Pan/Tilt/Zoom, rotation/tilt/zoom) includes the rotation, tilt and zoom information of the pan/tilt, which represents the omni-directional (left/right/up and down) movement of the pan/tilt and the zoom and zoom control of the lens.
  • the UAV projection device can communicate with the UAV flight control system through wireless transmission methods (such as Bluetooth, WiFi, 4G, 5G, etc.), and receive positioning information, attitude information, PTZ information, and UAV The focal length information of the camera and the video stream captured by the drone camera. After receiving the video stream returned by the drone, the video stream is decoded through soft decoding or hard decoding, and the video frame is obtained.
  • wireless transmission methods such as Bluetooth, WiFi, 4G, 5G, etc.
  • the UAV's positioning information, posture information, and PTZ information determine the initial position and posture of the UAV camera in the world coordinate system. Combining the UAV's positioning information, posture information, and PTZ information can be used as the approximate position and posture of the UAV camera in the world coordinate system (ie, the initial position and posture).
  • the three-dimensional map data is obtained, and the position, posture and focal length of the camera in the three-dimensional map are determined based on the initial position and posture and the corresponding focal length information, and the three-dimensional map is performed Rendering to obtain a two-dimensional picture corresponding to the picture taken by the camera at the initial position and posture.
  • the area of the two-dimensional picture should be larger than the area of the corresponding video frame, that is, the two-dimensional picture should cover the video frame.
  • the position of the camera can be determined according to the initial position and posture, and the three-dimensional map can be rendered in a bird's-eye view to obtain a two-dimensional picture.
  • the subsequent feature matching of the video frame and the two-dimensional picture can solve the problem caused by different perspectives. difference.
  • it is the real-time positioning information and attitude information of the UAV and the previous video frame to calculate the camera position and perform fusion filtering to obtain the initial position and attitude of the new frame, and render the three-dimensional map based on this initial position and attitude.
  • the three-dimensional map is established based on the world coordinate system. Even if there is an error between the coordinate system of the three-dimensional map and the world coordinate system, the offset caused by the error is within the error range (usually within a few meters). Rendering a three-dimensional map is not a big problem, and the rendered two-dimensional picture can still cover the target area (the area corresponding to the video frame).
  • S102 Perform feature matching on the video frame taken by the drone and the two-dimensional picture, and determine the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result.
  • feature matching is performed on the video frame taken by the drone camera and the two-dimensional picture, that is, the feature points in the video frame and the two-dimensional picture are extracted, and the feature points are determined according to the feature points.
  • the similarity (feature vector distance) between the two is matched, and the matching result is generated.
  • the feature points on the video frame are two-dimensional feature points
  • the feature points on the two-dimensional screen are matching feature points.
  • the correspondence between the three-dimensional map and the two-dimensional screen coordinate points can be recorded, and the three-dimensional features in the three-dimensional map corresponding to the matching feature points in the two-dimensional screen can be determined according to the record.
  • Point coordinates are used to determine the three-dimensional feature points corresponding to the two-dimensional feature points on the video frame on the three-dimensional map.
  • the RGB image (two-dimensional picture) and the depth map are simultaneously rendered when the two-dimensional picture is rendered.
  • the three-dimensional feature points corresponding to the points in the two-dimensional picture can be inversely calculated to determine the two-dimensional image on the video frame.
  • the feature point corresponds to the three-dimensional feature point on the three-dimensional map.
  • S103 Based on the two-dimensional feature points and the three-dimensional feature points, determine a camera pose matrix, focal length information, and/or distortion parameters through a pose solving algorithm.
  • the two-dimensional feature points and the three-dimensional feature points are substituted into the pose solving algorithm to obtain the camera pose matrix, focal length information, and/or Distortion parameters.
  • the pose solution algorithm is a method to solve the motion of 3D to 2D point pairs. It describes how to determine the pose, focal length and distortion of the camera when the n 3D space points and their projection positions are known.
  • whether the camera is distorted can be determined according to the specific type or parameters of the camera. For cameras with no distortion or less severe distortion, the distortion parameters can be set as default parameters (for example, if set to 0, the default camera has no distortion).
  • S104 Set a camera in the virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
  • the pose matrix and focal length information are input into the camera parameters of the virtual scene, and the position, posture and focal length of the camera are set. Then the video frame is added to the rendering pipeline, and the rendering pipeline performs real-time fusion projection of the video frame under the corresponding camera parameter settings.
  • the mapped overlap area is processed for smooth transition, thereby fusing the video frame into the three-dimensional scene, and completing the video projection of the corrected video frame in the three-dimensional scene. It is understandable that the video projection of the video frame can be performed based on the existing video projection method, which will not be repeated here.
  • the UAV under the initial position and posture is obtained.
  • the pose calculation algorithm can determine the camera pose matrix, focal length information and/or distortion parameters, set the camera in the virtual scene based on the above information, and add the video frame to the rendering pipeline for video projection, thereby realizing automatic real-time projection without work
  • the personnel manually fit the video no need to temporarily set up a surveillance gun on the scene, just put the drone on the scene, effectively respond to emergency situations, and match the video frame with the two-dimensional picture so that the video frame can be projected on the correct three-dimensional model In terms of location, it effectively improves the video projection effect.
  • Fig. 2 is a flowchart of another drone projection method provided by an embodiment of the application.
  • the drone projection method embodies the drone projection method.
  • the UAV projection method includes:
  • S201 Determine the initial position and attitude of the drone's camera in the world coordinate system based on the positioning information, attitude information, and pan/tilt information of the drone.
  • S202 Acquire a three-dimensional model tile corresponding to the rendering range according to the initial position and posture.
  • the three-dimensional map is stored in the form of three-dimensional model tiles.
  • the data volume of the three-dimensional map is relatively large.
  • each three-dimensional map is called a three-dimensional model tile, and the location range corresponding to each three-dimensional model tile is recorded. Determine and retrieve the 3D model tiles corresponding to the rendering range according to the initial position and posture.
  • the range of the three-dimensional map composed of the acquired three-dimensional model tiles should be larger than the range of the video frame captured by the camera.
  • a circle with a preset radius is determined by taking the vertical projection position of the camera as the origin, and all three-dimensional model tiles that intersect the circle correspond to the rendering range.
  • S203 Render the three-dimensional model tiles to obtain a two-dimensional picture corresponding to the initial position and posture.
  • the position and posture of the camera are set as the initial position and posture, and these 3D model tiles are rendered by the GPU visualization engine to obtain a two-dimensional picture corresponding to the initial posture. It can be understood that the display range of the two-dimensional picture is larger than the display range of the corresponding video frame.
  • S204 Acquire the initial position and posture and the time stamp of the video frame, and perform time synchronization between the initial position and posture and the video frame according to the correspondence between the time stamps.
  • time stamps are synchronized, and the time stamp of the initial position and attitude can also be determined based on the above information.
  • the initial position and posture are compared with the time stamp of the video frame, and when the time stamps are consistent, the corresponding initial position and posture, two-dimensional picture, and video frame are time synchronized.
  • S205 Obtain video frames taken by the drone and feature points and descriptors of the two-dimensional picture based on the image feature extraction algorithm.
  • image feature extraction is performed based on GPU, and image features include feature points and descriptors.
  • the image feature extraction algorithm can be SIFT (Scale Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Features, accelerated robust feature) algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, etc. This embodiment does not Make a limit.
  • the feature points of the image are some of the most representative points on the image.
  • the so-called most representative means that these points contain most of the information expressed by the image. Even if you rotate, zoom, or even adjust the brightness of the image, these points still exist steadily and will not be lost. Finding these points is equivalent to confirming the image, and they can be used for meaningful work such as matching and recognition.
  • Feature points are composed of two parts: Key-point and Descriptor.
  • the BRIEF descriptor is a binary descriptor, usually a 128-bit binary string. Its calculation method is to randomly select 128 point pairs from around the key point p. For two points in each point pair, if the gray value of the previous point is greater than that of the next point, then it will be 1, otherwise it will be 0.
  • extracting ORB features actually includes two things: extracting key points and calculating descriptors.
  • Use FAST feature point detection algorithm or Harris corner detection algorithm or SIFT, SURF and other algorithms to detect the location of feature points, and then in the feature point neighborhood Use the BRIEF algorithm to establish feature descriptors.
  • S206 Perform feature matching on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
  • the similarity between the two feature points is judged according to the distance of the corresponding descriptor, and the smaller the distance, the higher the similarity.
  • the distance of the descriptor can be Euclidean distance, Hamming distance, cosine distance, etc.
  • the feature points are sorted according to distance, and the matching results of the first N features are displayed under a certain degree of confidence, that is, the two-dimensional The feature points between the picture and the video frame are matched.
  • the RANSAC Random Sample Consensus
  • the basic matrix and homography matrix of the two-dimensional picture and video frame are obtained, and the matched feature points are screened by the RANSAC algorithm based on the basic matrix and the homography matrix to eliminate the feature points with matching errors.
  • S208 Determine the coordinates of the matching feature point in the two-dimensional picture that matches the two-dimensional feature point on the video frame according to the feature matching result.
  • the feature points that match each other in the two-dimensional picture and the video frame are respectively defined as the matching feature point and the two-dimensional feature point.
  • the matching feature points in the two-dimensional picture that match the two-dimensional feature points on the video frame are determined, and the coordinates of these matching feature points are determined.
  • the correspondence between the three-dimensional map and the coordinate points of the two-dimensional screen can be recorded.
  • the matching feature points corresponding to the two-dimensional feature points are determined according to the matching result, and then the corresponding relationship between the three-dimensional map and the two-dimensional screen coordinate points is obtained and matched The coordinates of the three-dimensional feature point corresponding to the feature point.
  • the RGB image (two-dimensional image) and the depth map are simultaneously rendered when the two-dimensional image is rendered.
  • the three-dimensional feature points corresponding to the points in the two-dimensional image can be inversely calculated, and the matching features on the two-dimensional image can be obtained.
  • the coordinates of the three-dimensional feature point corresponding to the point can be obtained.
  • S210 Obtain the two-dimensional feature point coordinates on the video frame and the three-dimensional feature point coordinates on the three-dimensional map, and substitute the two-dimensional feature point coordinates and the three-dimensional feature point coordinates into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera Pose matrix, focal length information and/or distortion parameters.
  • the PnP (Perspective-n-Point) algorithm is a method for solving 3D to 2D point pair motion, which can be solved by P3P, direct linear transformation (DLT), EPnP and other algorithms.
  • P3P algorithm is a 3D-2D pose solution method. It needs to know the matching 3D point and the image 2D point, that is, first find the corresponding 2D point (equivalent to the two-dimensional feature of this scheme Point) 3D coordinates in the current camera coordinate system (equivalent to the three-dimensional feature point coordinates of this solution), and then calculate the camera pose according to the 3D coordinates in the world coordinate system and the 3D coordinates in the current camera coordinate system.
  • the nonlinear optimization algorithm is a method of further optimizing the reprojection error of the 3D point to the 2D point through the least squares given the initial value of the camera posture focal length. During the optimization process, the algorithm will further refine the camera posture and focal length. Adjustment.
  • the LM (Levenberg-Marquardt) optimization algorithm is used as a non-linear optimization algorithm to optimize the initial value of the camera posture focal length to obtain the smallest reprojection error from 3D point to 2D point.
  • the two-dimensional feature point coordinates on the video frame and the three-dimensional feature point coordinates on the three-dimensional map are obtained, the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm, and the accurate solution is obtained by the PnP algorithm.
  • the camera pose matrix and then optimize the camera parameters through a nonlinear optimization algorithm to obtain focal length information and/or distortion parameters.
  • the distortion parameters can be determined according to the specific types or parameters of the camera. For cameras with no distortion or less severe distortion, the distortion parameters can be set as default parameters (for example, if set to 0, the default camera has no distortion), and the distortion parameters can be omitted. calculate.
  • S211 Set a camera in the virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
  • the UAV under the initial position and posture is obtained.
  • the two-dimensional picture corresponding to the range captured by the camera then feature matching between the two-dimensional picture and the video frame captured by the drone camera, and after the matching is completed, the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame is determined in the three-dimensional map.
  • the pose calculation algorithm can determine the camera pose matrix, focal length information and/or distortion parameters, set the camera in the virtual scene based on the above information, and add the video frame to the rendering pipeline for video projection, thereby realizing automatic real-time projection without work
  • the personnel manually fit the video no need to temporarily set up a surveillance camera on the spot, just put the drone on the spot, effectively respond to emergency situations, and determine the precise position of the camera based on image feature matching, PnP algorithm and nonlinear optimization algorithm, so that The video frame can be projected on the correct position of the 3D model, effectively improving the video projection effect.
  • Two-dimensional images are rendered in the form of three-dimensional model tiles, which reduces the burden of GPU graphics processing and effectively improves the real-time performance of video projection.
  • this solution does not require drones to be equipped with high-precision GPS, IMU and calibrated cameras. It is low-cost, easy to deliver, and easy to use. It can ensure that the video is projected on the correct position of the 3D model and can be put into emergency use in a timely and effective manner. Scene.
  • FIG. 3 is a schematic structural diagram of an unmanned aerial vehicle projection device provided by an embodiment of the application.
  • the drone projection device provided in this embodiment includes a two-dimensional rendering module 31, a feature corresponding module 32, a pose determination module 33 and a video projection module 34.
  • the two-dimensional rendering module 31 is used to determine the initial position and posture of the drone's camera in the world coordinate system based on the positioning information, attitude information, and pan-tilt information of the drone, and to render the three-dimensional map to obtain the The two-dimensional picture corresponding to the initial position and posture;
  • the feature corresponding module 32 is used to perform feature matching between the video frame taken by the drone and the two-dimensional picture, and determine the two-dimensional feature point on the video frame according to the result of the feature matching
  • the pose determination module 33 is used to determine the camera pose matrix, focal length information, and/or the camera pose based on the two-dimensional feature points and the three-dimensional feature points through a pose solving algorithm Distortion parameter;
  • a video projection module 34 configured to set a camera in a virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
  • the UAV under the initial position and posture is obtained.
  • the two-dimensional picture corresponding to the range captured by the camera then feature matching between the two-dimensional picture and the video frame captured by the drone camera, and after the matching is completed, the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame is determined in the three-dimensional map.
  • the pose calculation algorithm can determine the camera pose matrix, focal length information and/or distortion parameters, set the camera in the virtual scene according to the above information, and add the video frame to the rendering pipeline for video projection, thereby realizing automatic real-time projection without work
  • the personnel manually fit the video no need to temporarily set up a surveillance gun on the scene, just put the drone on the scene, effectively respond to emergency situations, and match the video frame with the two-dimensional picture so that the video frame can be projected on the correct three-dimensional model In terms of location, it effectively improves the video projection effect.
  • the two-dimensional rendering module 31 is specifically configured to determine the initial position and posture of the drone camera in the world coordinate system based on the positioning information, attitude information, and pan-tilt information of the drone;
  • the initial position and posture acquires a three-dimensional model tile corresponding to the rendering range, the three-dimensional map is stored in the form of a three-dimensional model tile; the three-dimensional model tile is rendered to obtain a two-dimensional picture corresponding to the initial position and posture .
  • the feature corresponding module 32 when the feature corresponding module 32 performs feature matching between the video frame taken by the drone and the two-dimensional picture, it specifically includes: obtaining the video frame taken by the drone based on an image feature extraction algorithm And feature points and descriptors of the two-dimensional picture; feature points between the video frame and the two-dimensional picture are matched according to the distance of the descriptor.
  • the feature corresponding module 32 after the feature corresponding module 32 performs feature matching on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor, it further compares the matched feature points based on the RANSAC algorithm. To filter.
  • it further includes a time synchronization module, which is used to obtain the initial position and posture and the all-dimensional image before the feature-corresponding module 32 performs feature matching between the video frame captured by the drone and the two-dimensional picture.
  • the time stamp of the video frame; and the time synchronization of the initial position and posture with the video frame is performed according to the correspondence between the time stamps.
  • the feature corresponding module 32 determines, according to the feature matching result, the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame on the three-dimensional map, it specifically includes: according to the feature matching result Determine the coordinates of the matching feature points in the two-dimensional image that match the two-dimensional feature points on the video frame; determine the corresponding two-dimensional feature points in the three-dimensional map according to the correspondence between the three-dimensional map and the two-dimensional image coordinate points The coordinates of the 3D feature point.
  • the pose determination module 33 is specifically configured to: obtain two-dimensional feature point coordinates on a video frame and three-dimensional feature point coordinates on a three-dimensional map; The three-dimensional feature point coordinates are substituted into the PnP algorithm and the non-linear optimization algorithm to obtain the camera pose matrix, focal length information and/or distortion parameters.
  • Fig. 4 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
  • the computer equipment includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs It is executed by the one or more processors 41, so that the one or more processors 41 implement the drone projection method provided in the foregoing embodiment.
  • the input device 43, the output device 44, the memory 42, and the processor 41 may be connected by a bus or other methods. In FIG. 4, the connection by a bus is taken as an example.
  • the memory 42 can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the drone projection method described in any embodiment of this application (for example, unmanned The two-dimensional rendering module 31, the feature corresponding module 32, the pose determination module 33 and the video projection module 34 in the camera projection device.
  • the memory 42 may mainly include a program storage area and a data storage area.
  • the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, and the like.
  • the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the memory 42 may further include a memory remotely provided with respect to the processor 41, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the input device 43 can be used to receive inputted digital or character information, and generate key signal input related to user settings and function control of the device.
  • the output device 44 may include a display device such as a display screen.
  • the processor 41 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 42 to realize the above-mentioned drone projection method.
  • the drone projection device and computer provided above can be used to execute the drone projection method provided by any of the above embodiments, and have corresponding functions and beneficial effects.
  • the embodiment of the present application also provides a storage medium containing computer-executable instructions, when the computer-executable instructions are executed by a computer processor, they are used to execute the drone projection method provided in the above-mentioned embodiments, and the drone projection
  • the method includes: determining the initial position and posture of the UAV camera in the world coordinate system based on the UAV's positioning information, posture information and PTZ information, and rendering a three-dimensional map to obtain a two-dimensional corresponding to the initial position and posture Picture; feature matching of the video frame taken by the drone and the two-dimensional picture, and determine the corresponding three-dimensional feature point of the two-dimensional feature point on the video frame on the three-dimensional map based on the feature matching result;
  • the two-dimensional feature points and the three-dimensional feature points are used to determine the camera pose matrix, focal length information and/or distortion parameters through a pose solving algorithm; according to the camera pose matrix, the focal length information and/or the distortion parameters
  • the camera in the virtual scene is set, and the video frame is added to
  • Storage medium any of various types of storage devices or storage devices.
  • the term "storage medium” is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc. ; Non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements.
  • the storage medium may further include other types of memory or a combination thereof.
  • the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the Internet).
  • the second computer system can provide the program instructions to the first computer for execution.
  • storage media may include two or more storage media that may reside in different locations (for example, in different computer systems connected through a network).
  • the storage medium may store program instructions (for example, embodied as a computer program) executable by one or more processors.
  • the storage medium containing computer-executable instructions provided by the embodiments of the present application is not limited to the drone projection method described above, and can also execute the unmanned aerial vehicle provided by any embodiment of the present application. Related operations in the projector projection method.
  • the drone projection device, equipment, and storage medium provided in the above embodiments can execute the drone projection method provided in any embodiment of this application.

Abstract

本申请实施例公开了一种无人机投影方法、装置、设备及存储介质。本申请实施例提供的技术方案通过基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头的初始位置姿态，并基于该初始位置姿态对三维地图进行渲染，获得与初始位置姿态下的无人机摄像头拍摄的范围对应的二维画面，然后对二维画面和无人机摄像头拍摄的视频帧进行特征匹配，匹配完成后确定三维地图中与视频帧上的二维特征点对应的三维特征点，通过位姿求解算法可确定相机位姿矩阵、焦距信息和/或畸变参数，根据以上信息设置虚拟场景中的相机，并将视频帧加入渲染管线中进行视频投影，从而实现自动实时投影，无需工作人员手动贴合视频。

Description

一种无人机投影方法、装置、设备及存储介质 技术领域
本申请实施例涉及图像处理领域，尤其涉及一种无人机投影方法、装置、设备及存储介质。
背景技术
视频投影技术是把监控视频和三维模型相结合，将关注区域的监控视频投影到大场景的三维模型中，可以实现静态大场景与动态重点场景的虚实结合。
传统的视频投影方案都是基于固定点位的视频监控枪机，即认为无人机摄像头的位置、姿态是固定的，在投影配置过程中，通过人工设置相机的视场角和位置姿态，使得相机在三维数字空间中的相对位置姿态和其在物理世界里的位置姿态相同，实现视频投影画面和三维模型的贴合。
然而在现场缺乏固定点位的视频监控枪机，或者枪机视角不好无法覆盖整个场景，或者该枪机并没有配置过视频投影的情况下，需要工作人员花费大量的时间去配置相机位置姿态等信息，存在无法及时满足视频投影要求的情况。
发明内容
本申请实施例提供一种无人机投影方法、装置、设备及存储介质，以满足视频投影的实时性需求。
在第一方面，本申请实施例提供了一种无人机投影方法，包括：
基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面；
对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点；
基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数；
根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
进一步的，所述基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面，包括：
基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态；
根据所述初始位置姿态获取渲染范围对应的三维模型瓦片，所述三维地图通过三维模型瓦片的形式进行存储；
对所述三维模型瓦片进行渲染以获得与所述初始位置姿态对应的二维画面。
进一步的，所述对无人机拍摄的视频帧和所述二维画面进行特征匹配，包括：
基于图像特征提取算法获取无人机拍摄的视频帧和所述二维画面的特征点以及描述子；
根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配。
进一步的，所述根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配之后，还包括：
基于RANSAC算法对匹配的特征点进行筛选。
进一步的，所述对无人机拍摄的视频帧和所述二维画面进行特征匹配之前，还包括：
获取所述初始位置姿态和所述视频帧的时间戳；
根据时间戳的对应关系对所述初始位置姿态和所述视频帧进行时间同步。
进一步的，所述根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点，包括：
根据特征匹配结果确定所述二维画面中与所述视频帧上的二维特征点匹配的匹配特征点的坐标；
根据三维地图和二维画面坐标点的对应关系，确定三维地图中与二维特征点对应的三维特征点的坐标。
进一步的，所述基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数，包括：
获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标；
将所述二维特征点坐标和所述三维特征点坐标代入PnP算法和非线性优化算法，以得到相机位姿矩阵、焦距信息和/或畸变参数。
在第二方面，本申请实施例提供了一种无人机投影装置，包括二维渲染模块、特征对应模块、位姿确定模块和视频投影模块，其中：
二维渲染模块，用于基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面；
特征对应模块，用于对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点；
位姿确定模块，用于基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数；
视频投影模块，用于根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
进一步的，所述二维渲染模块具体用于：
基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态；
根据所述初始位置姿态获取渲染范围对应的三维模型瓦片，所述三维地图通过三维模型瓦片的形式进行存储；
对所述三维模型瓦片进行渲染以获得与所述初始位置姿态对应的二维画面。
进一步的，所述特征对应模块在对无人机拍摄的视频帧和所述二维画面进行特征匹配时，具体包括：
基于图像特征提取算法获取无人机拍摄的视频帧和所述二维画面的特征点以及描述子；
根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配。
进一步的，所述特征对应模块在根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配之后，还基于RANSAC算法对匹配的特征点进行筛选。
进一步的，还包括时间同步模块，用于在所述特征对应模块对无人机拍摄 的视频帧和所述二维画面进行特征匹配之前，获取所述初始位置姿态和所述视频帧的时间戳；根据时间戳的对应关系对所述初始位置姿态和所述视频帧进行时间同步。
进一步的，所述特征对应模块在根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点时，具体包括：
根据特征匹配结果确定所述二维画面中与所述视频帧上的二维特征点匹配的匹配特征点的坐标；
根据三维地图和二维画面坐标点的对应关系，确定三维地图中与二维特征点对应的三维特征点的坐标。
进一步的，所述位姿确定模块具体用于：
获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标；
将所述二维特征点坐标和所述三维特征点坐标代入PnP算法和非线性优化算法，以得到相机位姿矩阵、焦距信息和/或畸变参数。
在第三方面，本申请实施例提供了一种计算机设备，包括：存储器以及一个或多个处理器；
所述存储器，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如第一方面所述的无人机投影方法。
在第四方面，本申请实施例提供了一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如第一方面所述的无人机投影方法。
本申请实施例通过先基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头的初始位置姿态，并基于该初始位置姿态对三维地图进行渲染，获得与初始位置姿态下的无人机摄像头拍摄的范围对应的二维画面，然后对二维画面和无人机摄像头拍摄的视频帧进行特征匹配，匹配完成后确定三维地图中与视频帧上的二维特征点对应的三维特征点，通过位姿求解算法可确定相机位姿矩阵、焦距信息和/或畸变参数，根据以上信息设置虚拟场景中的相机，并将视频帧加入渲染管线中进行视频投影，从而实现自动实时投影，无需工作人员手动贴合视频，不需要在现场临时设置监控枪机，在现场投入无人机即可，有效应对紧急情况，并且通过视频帧和二维画面的匹配使得视频帧可投影在三维模型的正确位置上，有效提高视频投影效果。
附图说明
图1是本申请实施例提供的一种无人机投影方法的流程图；
图2是本申请实施例提供的另一种无人机投影方法的流程图；
图3是本申请实施例提供的一种无人机投影装置的结构示意图；
图4是本申请实施例提供的一种计算机设备的结构示意图。
具体实施方式
为了使本申请的目的、技术方案和优点更加清楚，下面结合附图对本申请具体实施例作进一步的详细描述。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部内容。在更加详细地讨论示例性实施例之前应当提到的是，一些示例性实施例被描述成作为流程图描绘的处理或方法。虽然流程图将各项操作(或步骤)描述成顺序的处理，但是其中的许多操作可以被并行地、并发地或者同时实施。此外，各项操作的顺序可以被重新安排。当其操作完成时所述处理可以被终止，但是还可以具有未包括在附图中的附加步骤。所述处理可以对应于方法、函数、规程、子例程、子程序等等。
图1给出了本申请实施例提供的一种无人机投影方法的流程图，本申请实施例提供的无人机投影方法可以由无人机投影装置来执行，该无人机投影装置可以通过硬件和/或软件的方式实现，并集成在计算机设备中。
下述以无人机投影装置执行无人机投影方法为例进行描述。参考图1，该无人机投影方法包括：
S101：基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面。
可以理解的是，摄像头搭载在云台上，并且云台搭载在无人机上。其中，定位信息包括经度、纬度和高度，可通过无人机上搭载的GPS模块进行检测得出。姿态信息包括俯仰角、横滚角和偏航角，可通过无人机上搭载的IMU(惯性测量单元)检测得出。云台信息(PTZ，Pan/Tilt/Zoom，旋转/俯仰/变焦)包括云台的旋转、俯仰和变焦信息，代表云台全方位(左右/上下)移动及镜头变 倍、变焦控制。
示例性的，无人机投影装置可通过无线传输方式(例如蓝牙、WiFi、4G、5G等)与无人机飞控系统进行通讯，并接收定位信息、姿态信息、云台信息、无人机摄像头的焦距信息以及无人机摄像头拍摄的视频流。在接收到无人机回传的视频流后，经过软解码或硬解码的方式对视频流进行解码，并获得视频帧。
进一步的，基于无人机的定位信息、姿态信息和云台信息，确定无人机摄像头在世界坐标系下的初始位置姿态。将无人机的定位信息、姿态信息和云台信息进行合并即可作为无人机摄像头在世界坐标系下的大概位置姿态(即初始位置姿态)。
在确定无人机摄像头在世界坐标系下的初始位置姿态后，获取三维地图数据，并基于该初始位置姿态以及对应的焦距信息确定在三维地图中的相机的位置姿态以及焦距，对三维地图进行渲染，从而获取与摄像头在初始位置姿态拍摄的画面对应二维画面。优选的，二维画面的面积应大于对应视频帧的面积，即二维画面应覆盖视频帧。
在其他实施例中，可根据初始位置姿态确定相机的位置，并以俯视的姿态对三维地图进行渲染，得到二维画面，后续对视频帧和二维画面的特征匹配可解决不同视角带来的差异。或者是无人机的实时定位信息和姿态信息和前一视频帧解算出相机位置做融合滤波得到新一帧的初始位置姿态，并基于这个初始位置姿态对三维地图进行渲染。
可以理解的是，三维地图是基于世界坐标系建立的，即使三维地图的坐标系与世界坐标系存在误差，但是误差造成的偏移在误差范围内(一般在几米内)，这个偏移量对于渲染三维地图来说问题不大，渲染出的二维画面还是可以覆盖到目标区域(视频帧对应的区域)。
S102：对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点。
示例性的，在得到渲染出的二维画面后，对无人机摄像头拍摄的视频帧和二维画面进行特征匹配，即提取出视频帧和二维画面中的特征点，并根据特征点之间的相似度(特征向量距离)进行匹配，并生成匹配结果。其中，视频帧上的特征点为二维特征点，二维画面上的特征点为匹配特征点。
进一步的，在渲染二维画面的时候，可对三维地图和二维画面坐标点之间 的对应关系进行记录，并根据该记录确定三维地图中与二维画面中的匹配特征点对应的三维特征点坐标，确定视频帧上的二维特征点在三维地图上对应的三维特征点。另外，在渲染出二维画面时同时渲染了RGB图像(二维画面)和深度图，根据深度图亦可反算出二维画面中的点对应的三维特征点，进而确定视频帧上的二维特征点在三维地图上对应的三维特征点。
S103：基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数。
示例性的，在得到二维特征点和三维特征点的位置坐标和对应关系后，将二维特征点和三维特征点代入位姿求解算法中，从而得到相机位姿矩阵、焦距信息和/或畸变参数。
其中位姿求解算法是求解3D到2D点对运动的方法，描述了当知道n个3D空间点以及它们的投影位置时，如何确定相机所在的位姿、焦距以及畸变的方法，即在已知世界坐标系下N个空间点的真实坐标以及这些空间点在图像上的投影，如何计算相机所在的位姿的解决方法。其中摄像头是否畸变可根据摄像头的具体类型或参数进行确定，对于无畸变或者畸变不严重的摄像头，畸变参数可设置为默认参数(如设置为0，默认摄像头无畸变)。
S104：根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
示例性的，在获得位姿矩阵和焦距信息后，将位姿矩阵和焦距信息输入到虚拟场景的相机参数中，对相机的位置姿态和焦距进行设置。然后将视频帧加入到渲染管线中，并由渲染管线在对应的相机参数设置下对视频帧进行实时融合投影。
在进行融合投影时，确定视频帧中的像素点与三维场景(虚拟场景)中的三维点之间的映射关系，并根据映射关系将视频帧在三维场景中进行颜色纹理映射，并对颜色纹理映射的重合区域进行平滑过渡处理，从而将视频帧融合在三维场景中，完成校正视频帧在三维场景中的视频投影。可以理解的是，对视频帧的视频投影基于现有的视频投影方法进行即可，在此不做赘述。
上述，通过先基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头的初始位置姿态，并基于该初始位置姿态对三维地图进行渲染，获得与初始位置姿态下的无人机摄像头拍摄的范围对应的二维画面，然后对二维画面和无人机摄像头拍摄的视频帧进行特征匹配，匹配完成后确定三维地图中与视频 帧上的二维特征点对应的三维特征点，通过位姿求解算法可确定相机位姿矩阵、焦距信息和/或畸变参数，根据以上信息设置虚拟场景中的相机，并将视频帧加入渲染管线中进行视频投影，从而实现自动实时投影，无需工作人员手动贴合视频，不需要在现场临时设置监控枪机，在现场投入无人机即可，有效应对紧急情况，并且通过视频帧和二维画面的匹配使得视频帧可投影在三维模型的正确位置上，有效提高视频投影效果。
图2为本申请实施例提供的另一种无人机投影方法的流程图，该无人机投影方法是对无人机投影方法的具体化。参考图2，该无人机投影方法包括：
S201：基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态。
S202：根据所述初始位置姿态获取渲染范围对应的三维模型瓦片。
具体的，三维地图通过三维模型瓦片的形式进行存储。三维地图的数据量比较大，通过对三维地图数据进行切片，每片三维地图称为三维模型瓦片，并对每个三维模型瓦片对应的位置范围进行记录。根据初始位置姿态确定并调取渲染范围所对应的三维模型瓦片。
可以理解的是，由获取的三维模型瓦片组成的三维地图的范围应大于摄像头拍摄的视频帧的范围。例如以相机的垂直投影位置为原点确定一个预设半径的圆，与该圆存在交集的三维模型瓦片均与渲染范围对应。
S203：对所述三维模型瓦片进行渲染以获得与所述初始位置姿态对应的二维画面。
具体的，在获取与渲染范围对应的三维模型瓦片后，将相机的位置姿态设置为初始位置姿态，通过GPU可视化引擎对这些三维模型瓦片进行渲染并获得与初始姿态对应二维画面。可以理解的是，二维画面的显示范围大于对应视频帧的显示范围。
S204：获取所述初始位置姿态和所述视频帧的时间戳，并根据时间戳的对应关系对所述初始位置姿态和所述视频帧进行时间同步。
具体的，飞控系统在生成定位信息、姿态信息和云台信息时会同步打上时间戳，初始位置姿态的时间戳也可以基于以上信息进行确定。在确定初始位置姿态后，将初始位置姿态和视频帧的时间戳进行比对，在时间戳一致时，将对应的初始位置姿态、二维画面和视频帧进行时间同步。
S205：基于图像特征提取算法获取无人机拍摄的视频帧和所述二维画面的特征点以及描述子。
具体的，针对时间同步(同一时间戳)的视频帧和二维画面，基于GPU进行图像特征提取，图像特征包括特征点和描述子。其中图像特征提取算法可以是SIFT(Scale Invariant Feature Transform，尺度不变特征变换)算法、SURF(Speeded Up Robust Features，加速稳健特征)算法或ORB(Oriented FAST and Rotated BRIEF)算法等，本实施例不做限定。
其中图像的特征点是图像上最具代表性的一些点，所谓最具代表性就是说这些点包含了图像表述的大部分信息。即使旋转、缩放，甚至调整图像的亮度，这些点仍然稳定地存在，不会丢失。找出这些点，就相当于确定了这张图像，它们可以用来做匹配、识别等等有意义的工作。特征点由关键点(Key-point)和描述子(Descriptor)两部分组成。BRIEF描述子是一种二进制描述子，通常为128位的二进制串。它的计算方法是从关键点p周围随机挑选128个点对，对于每个点对中的两个点，如果前一个点的灰度值大于后一个点，则取1，反之取0。
比如，提取ORB特征其实包括了提取关键点和计算描述子两件事情，利用FAST特征点检测算法或Harris角点检测算法或SIFT、SURF等算法检测特征点的位置，接下来在特征点邻域利用BRIEF算法建立特征描述子。
S206：根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配。
具体的，在得到二维画面和视频帧上的特征点后，根据对应描述子的距离判断两个特征点之间的相似度，距离越小则相似度越高。其中描述子的距离可以是欧氏距离、汉明距离、余弦距离等。
进一步的，基于GPU遍历二维画面和视频帧的描述子，根据距离对特征点进行排序，在一定的置信度之下显示前N个特征的匹配结果，即根据距离反映的相似度将二维画面和视频帧之间的特征点进行匹配。
S207：基于RANSAC算法对匹配的特征点进行筛选。
其中，RANSAC(Random Sample Consensus，随机抽样一致)算法用于消除错误匹配的点。在完成特征点的匹配后，获取二维画面和视频帧的基础矩阵和单应矩阵，并基于基础矩阵和单应矩阵用RANSAC算法对匹配的特征点进行筛选，消除匹配错误的特征点。
S208：根据特征匹配结果确定所述二维画面中与所述视频帧上的二维特征点匹配的匹配特征点的坐标。
其中，二维画面和视频帧中相互匹配的特征点分别定义为匹配特征点和二维特征点。
具体的，在完成特征点的匹配与筛选后，确定二维画面中与视频帧上的二维特征点匹配的匹配特征点，并确定这些匹配特征点的坐标。
S209：根据三维地图和二维画面坐标点的对应关系，确定三维地图中与二维特征点对应的三维特征点的坐标。
具体的，在渲染二维画面的时候，可对三维地图和二维画面坐标点之间的对应关系进行记录。在需要确定与二维特征点对应的三维特征点的坐标时，根据匹配结果确定与二维特征点对应的匹配特征点，再根据三维地图和二维画面坐标点之间的对应关系获取与匹配特征点对应的三维特征点的坐标。
另外，在渲染二维画面时同时渲染了RGB图像(二维画面)和深度图，根据深度图亦可反算出二维画面中的点对应的三维特征点，并得到与二维画面上匹配特征点对应的三维特征点的坐标。
S210：获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标，并将所述二维特征点坐标和所述三维特征点坐标代入PnP算法和非线性优化算法，以得到相机位姿矩阵、焦距信息和/或畸变参数。
其中，PnP(Perspective-n-Point)算法是求解3D到2D点对运动的方法，可以通过P3P、直接线性变换(DLT)、EPnP等算法进行求解。以P3P算法为例，P3P算法是一种由3D-2D的位姿求解方式，需要已知匹配的3D点和图像2D点，即先求出对应的2D点(相当于本方案的二维特征点)在当前相机坐标系下的3D坐标(相当于本方案的三维特征点坐标)，然后根据世界坐标系下的3D坐标和当前相机坐标系下的3D坐标求解相机位姿。
非线性优化算法是在给定相机姿态焦距初始值的情况下，通过最小二乘进一步优化3D点到2D点重投影误差的方法，在优化过程中算法会进一步对相机的姿态和焦距做细微的调整。例如，通过LM(Levenberg-Marquardt)优化算法作为非线性优化算法对相机姿态焦距初始值进行优化，以得到3D点到2D点最小的重投影误差。
具体的，获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标，将二维特征点坐标和三维特征点坐标代入PnP算法和非线性优化算法，经PnP 算法求解得到准确的相机位姿矩阵，然后通过非线性优化算法对相机参数进行优化得到焦距信息和/或畸变参数。畸变参数的确定可根据摄像头的具体类型或参数进行确定，对于无畸变或者畸变不严重的摄像头，畸变参数可设置为默认参数(如设置为0，默认摄像头无畸变)，并可不对畸变参数进行计算。
S211：根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
上述，通过先基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头的初始位置姿态，并基于该初始位置姿态对三维地图进行渲染，获得与初始位置姿态下的无人机摄像头拍摄的范围对应的二维画面，然后对二维画面和无人机摄像头拍摄的视频帧进行特征匹配，匹配完成后确定三维地图中与视频帧上的二维特征点对应的三维特征点，通过位姿求解算法可确定相机位姿矩阵、焦距信息和/或畸变参数，根据以上信息设置虚拟场景中的相机，并将视频帧加入渲染管线中进行视频投影，从而实现自动实时投影，无需工作人员手动贴合视频，不需要在现场临时设置监控枪机，在现场投入无人机即可，有效应对紧急情况，并基于图像特征匹配、PnP算法和非线性优化算法确定相机的精确位置，使得视频帧可投影在三维模型的正确位置上，有效提高视频投影效果。通过三维模型瓦片的形式渲染出二维画面，减少GPU图形处理的负担，有效提高视频投影的实时性。同时，本方案不需要无人机搭载高精度的GPS、IMU和经过标定的相机，成本低、易交付、易使用，并可保证视频被投影在三维模型的正确位置，及时有效地投入紧急使用的场景中。
图3为本申请实施例提供的一种无人机投影装置的结构示意图。参考图3，本实施例提供的无人机投影装置包括二维渲染模块31、特征对应模块32、位姿确定模块33和视频投影模块34。
其中，二维渲染模块31，用于基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面；特征对应模块32，用于对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点；位姿确定模块33，用于基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数；视频投影模块34，用于根据所述相机位姿矩阵、 所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
上述，通过先基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头的初始位置姿态，并基于该初始位置姿态对三维地图进行渲染，获得与初始位置姿态下的无人机摄像头拍摄的范围对应的二维画面，然后对二维画面和无人机摄像头拍摄的视频帧进行特征匹配，匹配完成后确定三维地图中与视频帧上的二维特征点对应的三维特征点，通过位姿求解算法可确定相机位姿矩阵、焦距信息和/或畸变参数，根据以上信息设置虚拟场景中的相机，并将视频帧加入渲染管线中进行视频投影，从而实现自动实时投影，无需工作人员手动贴合视频，不需要在现场临时设置监控枪机，在现场投入无人机即可，有效应对紧急情况，并且通过视频帧和二维画面的匹配使得视频帧可投影在三维模型的正确位置上，有效提高视频投影效果。
在一个可能的实施例中，所述二维渲染模块31具体用于：基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态；根据所述初始位置姿态获取渲染范围对应的三维模型瓦片，所述三维地图通过三维模型瓦片的形式进行存储；对所述三维模型瓦片进行渲染以获得与所述初始位置姿态对应的二维画面。
在一个可能的实施例中，所述特征对应模块32在对无人机拍摄的视频帧和所述二维画面进行特征匹配时，具体包括：基于图像特征提取算法获取无人机拍摄的视频帧和所述二维画面的特征点以及描述子；根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配。
在一个可能的实施例中，所述特征对应模块32在根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配之后，还基于RANSAC算法对匹配的特征点进行筛选。
在一个可能的实施例中，还包括时间同步模块，用于在所述特征对应模块32对无人机拍摄的视频帧和所述二维画面进行特征匹配之前，获取所述初始位置姿态和所述视频帧的时间戳；根据时间戳的对应关系对所述初始位置姿态和所述视频帧进行时间同步。
在一个可能的实施例中，所述特征对应模块32在根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点时，具体包括：根据特征匹配结果确定所述二维画面中与所述视频帧上的二维特征点匹配的匹 配特征点的坐标；根据三维地图和二维画面坐标点的对应关系，确定三维地图中与二维特征点对应的三维特征点的坐标。
在一个可能的实施例中，所述位姿确定模块33具体用于：获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标；将所述二维特征点坐标和所述三维特征点坐标代入PnP算法和非线性优化算法，以得到相机位姿矩阵、焦距信息和/或畸变参数。
本申请实施例还提供了一种计算机设备，该计算机设备可集成本申请实施例提供的无人机投影装置。图4是本申请实施例提供的一种计算机设备的结构示意图。参考图4，该计算机设备包括：输入装置43、输出装置44、存储器42以及一个或多个处理器41；所述存储器42，用于存储一个或多个程序；当所述一个或多个程序被所述一个或多个处理器41执行，使得所述一个或多个处理器41实现如上述实施例提供的无人机投影方法。其中输入装置43、输出装置44、存储器42和处理器41可以通过总线或者其他方式连接，图4中以通过总线连接为例。
存储器42作为一种计算设备可读存储介质，可用于存储软件程序、计算机可执行程序以及模块，如本申请任意实施例所述的无人机投影方法对应的程序指令/模块(例如，无人机投影装置中的二维渲染模块31、特征对应模块32、位姿确定模块33和视频投影模块34)。存储器42可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据设备的使用所创建的数据等。此外，存储器42可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储器42可进一步包括相对于处理器41远程设置的存储器，这些远程存储器可以通过网络连接至设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入装置43可用于接收输入的数字或字符信息，以及产生与设备的用户设置以及功能控制有关的键信号输入。输出装置44可包括显示屏等显示设备。
处理器41通过运行存储在存储器42中的软件程序、指令以及模块，从而执行设备的各种功能应用以及数据处理，即实现上述的无人机投影方法。
上述提供的无人机投影装置和计算机可用于执行上述任意实施例提供的无 人机投影方法，具备相应的功能和有益效果。
本申请实施例还提供一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如上述实施例提供的无人机投影方法，该无人机投影方法包括：基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面；对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点；基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数；根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
存储介质——任何的各种类型的存储器设备或存储设备。术语“存储介质”旨在包括：安装介质，例如CD-ROM、软盘或磁带装置；计算机系统存储器或随机存取存储器，诸如DRAM、DDR RAM、SRAM、EDO RAM，兰巴斯(Rambus)RAM等；非易失性存储器，诸如闪存、磁介质(例如硬盘或光存储)；寄存器或其它相似类型的存储器元件等。存储介质可以还包括其它类型的存储器或其组合。另外，存储介质可以位于程序在其中被执行的第一计算机系统中，或者可以位于不同的第二计算机系统中，第二计算机系统通过网络(诸如因特网)连接到第一计算机系统。第二计算机系统可以提供程序指令给第一计算机用于执行。术语“存储介质”可以包括可以驻留在不同位置中(例如在通过网络连接的不同计算机系统中)的两个或更多存储介质。存储介质可以存储可由一个或多个处理器执行的程序指令(例如具体实现为计算机程序)。
当然，本申请实施例所提供的一种包含计算机可执行指令的存储介质，其计算机可执行指令不限于如上所述的无人机投影方法，还可以执行本申请任意实施例所提供的无人机投影方法中的相关操作。
上述实施例中提供的无人机投影装置、设备及存储介质可执行本申请任意实施例所提供的无人机投影方法，未在上述实施例中详尽描述的技术细节，可参见本申请任意实施例所提供的无人机投影方法。
上述仅为本申请的较佳实施例及所运用的技术原理。本申请不限于这里所 述的特定实施例，对本领域技术人员来说能够进行的各种明显变化、重新调整及替代均不会脱离本申请的保护范围。因此，虽然通过以上实施例对本申请进行了较为详细的说明，但是本申请不仅仅限于以上实施例，在不脱离本申请构思的情况下，还可以包括更多其他等效实施例，而本申请的范围由权利要求的范围决定。

Claims (10)

  1. 一种无人机投影方法，其特征在于，包括：
    基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面；
    对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点；
    基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数；
    根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
  2. 根据权利要求1所述的无人机投影方法，其特征在于，所述基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面，包括：
    基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态；
    根据所述初始位置姿态获取渲染范围对应的三维模型瓦片，所述三维地图通过三维模型瓦片的形式进行存储；
    对所述三维模型瓦片进行渲染以获得与所述初始位置姿态对应的二维画面。
  3. 根据权利要求1所述的无人机投影方法，其特征在于，所述对无人机拍摄的视频帧和所述二维画面进行特征匹配，包括：
    基于图像特征提取算法获取无人机拍摄的视频帧和所述二维画面的特征点以及描述子；
    根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配。
  4. 根据权利要求3所述的无人机投影方法，其特征在于，所述根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配之后，还包括：
    基于RANSAC算法对匹配的特征点进行筛选。
  5. 根据权利要求3所述的无人机投影方法，其特征在于，所述对无人机拍摄的视频帧和所述二维画面进行特征匹配之前，还包括：
    获取所述初始位置姿态和所述视频帧的时间戳；
    根据时间戳的对应关系对所述初始位置姿态和所述视频帧进行时间同步。
  6. 根据权利要求1所述的无人机投影方法，其特征在于，所述根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点，包括：
    根据特征匹配结果确定所述二维画面中与所述视频帧上的二维特征点匹配的匹配特征点的坐标；
    根据三维地图和二维画面坐标点的对应关系，确定三维地图中与二维特征点对应的三维特征点的坐标。
  7. 根据权利要求1所述的无人机投影方法，其特征在于，所述基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数，包括：
    获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标；
    将所述二维特征点坐标和所述三维特征点坐标代入PnP算法和非线性优化算法，以得到相机位姿矩阵、焦距信息和/或畸变参数。
  8. 一种无人机投影装置，其特征在于，包括二维渲染模块、特征对应模块、位姿确定模块和视频投影模块，其中：
    二维渲染模块，用于基于无人机的定位信息、姿态信息和云台信息确定无人机摄像头在世界坐标系下的初始位置姿态，并对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面；
    特征对应模块，用于对无人机拍摄的视频帧和所述二维画面进行特征匹配，并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点；
    位姿确定模块，用于基于所述二维特征点和所述三维特征点，通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数；
    视频投影模块，用于根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机，并将所述视频帧加入渲染管线中进行视频投影。
  9. 一种计算机设备，其特征在于，包括：存储器以及一个或多个处理器；
    所述存储器，用于存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-7任一所述的无人机投影方法。
  10. 一种包含计算机可执行指令的存储介质，其特征在于，所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-7任一所述的无人机投影方法。
PCT/CN2020/121663 2020-05-14 2020-10-16 一种无人机投影方法、装置、设备及存储介质 WO2021227359A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010408954.1A CN111586360B (zh) 2020-05-14 2020-05-14 一种无人机投影方法、装置、设备及存储介质
CN202010408954.1 2020-05-14

Publications (1)

Publication Number Publication Date
WO2021227359A1 true WO2021227359A1 (zh) 2021-11-18

Family

ID=72110842

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/121663 WO2021227359A1 (zh) 2020-05-14 2020-10-16 一种无人机投影方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN111586360B (zh)
WO (1) WO2021227359A1 (zh)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114494984A (zh) * 2022-04-18 2022-05-13 四川腾盾科技有限公司 基于无人机航拍数据的随遇静目标三维重建与定位的方法
CN114779679A (zh) * 2022-03-23 2022-07-22 北京英智数联科技有限公司 一种增强现实巡检系统及方法
CN115065867A (zh) * 2022-08-17 2022-09-16 中国科学院空天信息创新研究院 基于无人机视频金字塔模型的动态处理方法及装置
CN115760964A (zh) * 2022-11-10 2023-03-07 亮风台（上海）信息科技有限公司 一种获取目标对象的屏幕位置信息的方法与设备
CN116012377A (zh) * 2023-03-24 2023-04-25 四川腾盾科技有限公司 一种基于卫星地图的无人机虚拟观测图像生成及定位方法
CN116758157A (zh) * 2023-06-14 2023-09-15 深圳市华赛睿飞智能科技有限公司 一种无人机室内三维空间测绘方法、系统及存储介质
CN116958271A (zh) * 2023-06-06 2023-10-27 阿里巴巴（中国）有限公司 标定参数确定方法以及装置
CN117115414A (zh) * 2023-10-23 2023-11-24 西安羚控电子科技有限公司 基于深度学习的无gps无人机定位方法及装置
CN117237438A (zh) * 2023-09-18 2023-12-15 共享数据(福建)科技有限公司 一种三维模型和无人机视频数据的范围匹配方法与终端

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111586360B (zh) * 2020-05-14 2021-09-10 佳都科技集团股份有限公司 一种无人机投影方法、装置、设备及存储介质
CN111640181A (zh) * 2020-05-14 2020-09-08 佳都新太科技股份有限公司 一种交互式视频投影方法、装置、设备及存储介质
CN112907671B (zh) * 2021-03-31 2022-08-02 深圳市慧鲤科技有限公司 点云数据生成方法、装置、电子设备及存储介质
CN113766119B (zh) * 2021-05-11 2023-12-05 腾讯科技（深圳）有限公司 虚拟形象显示方法、装置、终端及存储介质
CN113256718B (zh) * 2021-05-27 2023-04-07 浙江商汤科技开发有限公司 定位方法和装置、设备及存储介质
CN113415433B (zh) * 2021-07-30 2022-11-29 成都纵横大鹏无人机科技有限公司 基于三维场景模型的吊舱姿态修正方法、装置和无人机
CN113793379A (zh) * 2021-08-12 2021-12-14 视辰信息科技(上海)有限公司 相机姿态求解方法及系统、设备和计算机可读存储介质
CN115965753A (zh) * 2022-12-26 2023-04-14 应急管理部大数据中心 一种空地协同快速三维建模系统、电子设备及存储介质
CN116612184B (zh) * 2023-04-11 2023-12-05 西南交通大学 一种基于监控场景的无人机相机位姿精确估算方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103198488A (zh) * 2013-04-16 2013-07-10 北京天睿空间科技有限公司 Ptz监控摄像机实时姿态快速估算方法
CN107564098A (zh) * 2017-08-17 2018-01-09 中山大学 一种大区域网络三维噪声地图的快速渲染方法
CN109618134A (zh) * 2018-12-10 2019-04-12 北京智汇云舟科技有限公司 一种无人机动态视频三维地理信息实时融合系统及方法
CN109727288A (zh) * 2017-12-28 2019-05-07 北京京东尚科信息技术有限公司 用于单目同时定位与地图构建的系统和方法
US10515458B1 (en) * 2017-09-06 2019-12-24 The United States Of America, As Represented By The Secretary Of The Navy Image-matching navigation method and apparatus for aerial vehicles
CN110799921A (zh) * 2018-07-18 2020-02-14 深圳市大疆创新科技有限公司 拍摄方法、装置和无人机
CN111586360A (zh) * 2020-05-14 2020-08-25 佳都新太科技股份有限公司 一种无人机投影方法、装置、设备及存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2849150A1 (en) * 2013-09-17 2015-03-18 Thomson Licensing Method for capturing the 3D motion of an object, unmanned aerial vehicle and motion capture system
CN104457704B (zh) * 2014-12-05 2016-05-25 北京大学 基于增强地理信息的无人机地面目标定位系统及方法
US10186075B2 (en) * 2016-11-30 2019-01-22 Adcor Magnet Systems, Llc System, method, and non-transitory computer-readable storage media for generating 3-dimensional video images
CN109920048A (zh) * 2019-02-15 2019-06-21 北京清瞳时代科技有限公司 监控画面生成方法及装置
CN110047142A (zh) * 2019-03-19 2019-07-23 中国科学院深圳先进技术研究院 无人机三维地图构建方法、装置、计算机设备及存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103198488A (zh) * 2013-04-16 2013-07-10 北京天睿空间科技有限公司 Ptz监控摄像机实时姿态快速估算方法
CN107564098A (zh) * 2017-08-17 2018-01-09 中山大学 一种大区域网络三维噪声地图的快速渲染方法
US10515458B1 (en) * 2017-09-06 2019-12-24 The United States Of America, As Represented By The Secretary Of The Navy Image-matching navigation method and apparatus for aerial vehicles
CN109727288A (zh) * 2017-12-28 2019-05-07 北京京东尚科信息技术有限公司 用于单目同时定位与地图构建的系统和方法
CN110799921A (zh) * 2018-07-18 2020-02-14 深圳市大疆创新科技有限公司 拍摄方法、装置和无人机
CN109618134A (zh) * 2018-12-10 2019-04-12 北京智汇云舟科技有限公司 一种无人机动态视频三维地理信息实时融合系统及方法
CN111586360A (zh) * 2020-05-14 2020-08-25 佳都新太科技股份有限公司 一种无人机投影方法、装置、设备及存储介质

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114779679A (zh) * 2022-03-23 2022-07-22 北京英智数联科技有限公司 一种增强现实巡检系统及方法
CN114494984A (zh) * 2022-04-18 2022-05-13 四川腾盾科技有限公司 基于无人机航拍数据的随遇静目标三维重建与定位的方法
CN115065867A (zh) * 2022-08-17 2022-09-16 中国科学院空天信息创新研究院 基于无人机视频金字塔模型的动态处理方法及装置
CN115065867B (zh) * 2022-08-17 2022-11-11 中国科学院空天信息创新研究院 基于无人机视频金字塔模型的动态处理方法及装置
CN115760964B (zh) * 2022-11-10 2024-03-15 亮风台（上海）信息科技有限公司 一种获取目标对象的屏幕位置信息的方法与设备
CN115760964A (zh) * 2022-11-10 2023-03-07 亮风台（上海）信息科技有限公司 一种获取目标对象的屏幕位置信息的方法与设备
CN116012377A (zh) * 2023-03-24 2023-04-25 四川腾盾科技有限公司 一种基于卫星地图的无人机虚拟观测图像生成及定位方法
CN116958271A (zh) * 2023-06-06 2023-10-27 阿里巴巴（中国）有限公司 标定参数确定方法以及装置
CN116758157A (zh) * 2023-06-14 2023-09-15 深圳市华赛睿飞智能科技有限公司 一种无人机室内三维空间测绘方法、系统及存储介质
CN116758157B (zh) * 2023-06-14 2024-01-30 深圳市华赛睿飞智能科技有限公司 一种无人机室内三维空间测绘方法、系统及存储介质
CN117237438A (zh) * 2023-09-18 2023-12-15 共享数据(福建)科技有限公司 一种三维模型和无人机视频数据的范围匹配方法与终端
CN117115414B (zh) * 2023-10-23 2024-02-23 西安羚控电子科技有限公司 基于深度学习的无gps无人机定位方法及装置
CN117115414A (zh) * 2023-10-23 2023-11-24 西安羚控电子科技有限公司 基于深度学习的无gps无人机定位方法及装置

Also Published As

Publication number Publication date
CN111586360B (zh) 2021-09-10
CN111586360A (zh) 2020-08-25

Similar Documents

Publication Publication Date Title
WO2021227359A1 (zh) 一种无人机投影方法、装置、设备及存储介质
WO2021227360A1 (zh) 一种交互式视频投影方法、装置、设备及存储介质
TWI619088B (zh) 圖像資料處理系統和相關方法以及相關圖像融合方法
CN108932051B (zh) 增强现实图像处理方法、装置及存储介质
US9667862B2 (en) Method, system, and computer program product for gamifying the process of obtaining panoramic images
WO2018214365A1 (zh) 图像校正方法、装置、设备、系统及摄像设备和显示设备
US8355565B1 (en) Producing high quality depth maps
JP5740884B2 (ja) 繰り返し撮影用ａｒナビゲーション及び差異抽出のシステム、方法及びプログラム
WO2020014909A1 (zh) 拍摄方法、装置和无人机
WO2020063100A1 (zh) 增强现实的图像展示方法、装置及设备
WO2017020150A1 (zh) 一种图像处理方法、装置及摄像机
WO2021184302A1 (zh) 图像处理方法、装置、成像设备、可移动载体及存储介质
CN111625091B (zh) 基于ar眼镜的标签叠加方法及装置
US9756260B1 (en) Synthetic camera lenses
CN111127524A (zh) 一种轨迹跟踪与三维重建方法、系统及装置
WO2019037038A1 (zh) 图像处理方法、装置及服务器
US20220067974A1 (en) Cloud-Based Camera Calibration
WO2018216341A1 (ja) 情報処理装置、情報処理方法、及びプログラム
WO2021104308A1 (zh) 全景深度测量方法、四目鱼眼相机及双目鱼眼相机
JP2020053774A (ja) 撮像装置および画像記録方法
CN111669499B (zh) 一种视频防抖方法、装置及视频采集设备
CN111294580B (zh) 基于gpu的摄像头视频投影方法、装置、设备及存储介质
US20220405968A1 (en) Method, apparatus and system for image processing
US10198842B2 (en) Method of generating a synthetic image
WO2021185036A1 (zh) 点云数据的生成及实时显示方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20936046

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20936046

Country of ref document: EP

Kind code of ref document: A1