CN115826628A - NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method - Google Patents

NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method Download PDF

Info

Publication number
CN115826628A
CN115826628A CN202310147930.9A CN202310147930A CN115826628A CN 115826628 A CN115826628 A CN 115826628A CN 202310147930 A CN202310147930 A CN 202310147930A CN 115826628 A CN115826628 A CN 115826628A
Authority
CN
China
Prior art keywords
module
nerf
aerial vehicle
unmanned aerial
depth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310147930.9A
Other languages
Chinese (zh)
Other versions
CN115826628B (en
Inventor
周仁建
王聪
姚慧敏
张继花
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Aeronautic Polytechnic
Original Assignee
Chengdu Aeronautic Polytechnic
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Aeronautic Polytechnic filed Critical Chengdu Aeronautic Polytechnic
Priority to CN202310147930.9A priority Critical patent/CN115826628B/en
Publication of CN115826628A publication Critical patent/CN115826628A/en
Application granted granted Critical
Publication of CN115826628B publication Critical patent/CN115826628B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Image Analysis (AREA)

Abstract

The invention discloses a heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method based on a NeRF neural network, and belongs to the technical field of unmanned aerial vehicle visual obstacle avoidance. The invention solves the problem that the SLAM visual obstacle avoidance is unstable by adopting the traditional RGB-D depth camera under the condition that the unmanned aerial vehicle directly irradiates sunlight, and improves the reliability of autonomous flight of the unmanned aerial vehicle.

Description

NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method
Technical Field
The invention belongs to the technical field of unmanned aerial vehicle visual obstacle avoidance, and particularly relates to a heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method based on a NeRF neural network.
Background
NeRF (nerve radiation field) is one of the current areas of intense research, and the problem to be solved is how to generate new views from new perspectives given some captured views. The method is different from the traditional three-dimensional reconstruction method which represents the scene as point cloud, grid, voxel and other explicit expressions, and is a novel method, the scene is modeled into a continuous 5D radiation field and is implicitly stored in a neural network, only sparse multi-angle image training with position is needed to be input to obtain a neural radiation field model, and clear pictures at any visual angle can be rendered according to the model.
Fusing deep learning with traditional geometry is a trend in SLAM development. In the prior art, some single-point modules in SLAM are replaced by neural networks, such as feature extraction, feature matching, loop back and depth estimation. Compared with single-point replacement, the NeRF-based SLAM method is a brand-new framework and can replace the traditional SLAM from end to end, and both the design method and the implementation architecture are adopted.
In the autonomous flight process of the unmanned aerial vehicle, flexible obstacle avoidance needs to be achieved, and the method generally adopted is to perform SLAM map reconstruction based on a laser radar auxiliary RGB-D depth camera or a binocular camera.
Although many three-dimensional reconstruction solutions are based on RGB-D or lidar sensors, scene reconstruction from monocular images provides a more convenient solution. RGBD cameras may fail under certain conditions, such as direct sunlight, and the lidar is still heavier than a monocular RGB camera. In addition, stereo cameras reduce the depth estimation problem to a one-dimensional disparity search, but rely on accurate calibration of cameras that are prone to miscalibration in actual operation.
Disclosure of Invention
Aiming at the defects in the prior art, the heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method based on the NeRF neural network provided by the invention solve the problem that the SLAM visual obstacle avoidance is unstable by adopting the traditional RGB-D depth camera under the condition that the unmanned aerial vehicle is directly irradiated by sunlight.
In order to achieve the purpose of the invention, the invention adopts the technical scheme that: a heterogeneous unmanned aerial vehicle visual obstacle avoidance system based on a NeRF neural network comprises a core module, and a laser radar module, an unmanned aerial vehicle flight control module, an unmanned aerial vehicle monocular camera module and an IMU inertial navigation module which are respectively connected with the core module;
the unmanned aerial vehicle monocular camera module is used for collecting images to obtain original images;
the IMU inertial navigation module is used for acquiring the camera attitude;
the core module is used for obtaining a global map according to the original image and the camera posture;
the laser radar module is used for obtaining a path planning result by combining the global map and the core module;
and the unmanned aerial vehicle flight control module is used for controlling the unmanned aerial vehicle according to the path planning result to realize obstacle avoidance of the unmanned aerial vehicle.
The invention has the beneficial effects that: according to the invention, an unmanned aerial vehicle monocular camera module is used for collecting images, an image collecting subunit is arranged in an FPGA editable logic submodule of a core module, the collected images are transmitted to a NeRF-SLAM mapping subunit, and the NeRF-SLAM mapping subunit is combined with an IMU inertial navigation module and an attitude solution subunit, so that a scene is modeled into a continuous 5D radiation field, and SLAM mapping can be performed through a three-dimensional scene implicit stored in a neural network. Uploading the map building result to an APU application processor sub-module through a MicroBlaze processor unit to update a global map of an obstacle avoidance system, planning a flight path by combining a laser radar module and a laser radar SLAM unit, and sending the path planning result to an unmanned aerial vehicle flight control system to control an unmanned aerial vehicle so as to realize visual obstacle avoidance of the unmanned aerial vehicle; the monocular camera module of the unmanned aerial vehicle is used, so that the weight and the size of the unmanned aerial vehicle are reduced; meanwhile, the concept of heterogeneous treatment is adopted, the generalization capability of the NeRF-SLAM radiation nerve field is improved, and the NeRF-SLAM radiation nerve field cannot fall into a local minimum value.
Furthermore, the core module comprises an APU application processor submodule and an FPGA editable logic submodule connected with the APU application processor submodule;
the APU application processor sub-module comprises a laser radar SLAM unit, a global map unit and a flight path planning unit which are connected in sequence; the laser radar SLAM unit is connected with the laser radar module; the global map unit is connected with the FPGA editable logic sub-module; the flight path planning unit is connected with the unmanned aerial vehicle flight control module;
the FPGA editable logic sub-module comprises a MicroBlaze processor unit and a NeRF-SLAM radiating neural field unit which are sequentially connected; the NeRF-SLAM radiation nerve field unit comprises a NeRF-SLAM mapping subunit, an image acquisition subunit and an attitude calculation subunit, wherein the image acquisition subunit and the attitude calculation subunit are respectively connected with the NeRF-SLAM mapping subunit; the MicroBlaze processor unit is respectively connected with the NeRF-SLAM map building subunit and the global map unit; the image acquisition subunit is connected with the monocular camera module of the unmanned aerial vehicle; and the attitude resolving subunit is connected with the IMU inertial navigation module.
The beneficial effects of the above further scheme are: by adopting the idea of isomerism processing, the generalization capability of the NeRF-SLAM radiation neural field is improved, and the NeRF-SLAM radiation neural field cannot fall into a local minimum value.
The invention provides a NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance method, which comprises the following steps:
s1, obtaining an original image through a monocular camera module of an unmanned aerial vehicle;
s2, modeling the original image into a continuous 5D radiation field by using a NeRF-SLAM radiation nerve field unit to obtain a mapping result;
s3, transmitting the mapping result to a global map unit through a MicroBlaze processor unit to obtain a global map;
s4, obtaining a path planning result by utilizing a laser radar SLAM unit and a laser radar module according to the global map;
and S5, transmitting the path planning result to an unmanned aerial vehicle flight control module by using a flight path planning unit, and realizing the visual obstacle avoidance of the unmanned aerial vehicle.
The invention has the beneficial effects that: the structure of the Hessian matrix is used for optimizing the NeRF-SLAM model, meanwhile, the MLP multilayer perceptron in the classical NeRF network is replaced by the SLFNN (single hidden layer feedforward neural network), an ELM (extreme learning machine) model is conveniently constructed, solution is accelerated, and reliability of autonomous flight of the unmanned aerial vehicle is improved.
Further, the step S2 is specifically as follows:
s201, according to the original image, utilizing a NeRF-SLAM to radiate an image acquisition subunit of a neural field unit to obtain a continuous frame RGB image;
s202, optimizing a NeRF-SLAM model by utilizing the structure of a Hessian matrix according to the continuous frame RGB images to obtain an improved NeRF-SLAM model;
s203, obtaining a mapping result according to the improved NeRF-SLAM model.
The beneficial effects of the above further scheme are: the structure of the Hessian matrix is used for optimizing the NeRF-SLAM model, so that the radiation field parameters can be optimized, and the camera posture can be refined.
Further, the improved NeRF-SLAM model in step S202 includes a first single hidden layer feedforward neural network SLFNN1, a second single hidden layer feedforward neural network SLFNN2, a calculation and position coding module, a volume rendering module, and a GRU convolution module;
the calculation and position coding module is used for acquiring a scene position and an observation direction;
the first single hidden layer feedforward neural network SLFNN1 is used for obtaining voxel density and feature vectors according to the scene position;
the second single hidden layer feedforward neural network SLFNN2 is used for obtaining colors according to the observation direction and the characteristic vector;
the volume rendering module is used for obtaining voxel output according to the voxel density and the color;
and the GRU convolution module is used for acquiring the output of the GRU convolution module.
The beneficial effects of the above further scheme are: the MLP multilayer perceptron in the classical NeRF network is replaced by an SLFNN (single hidden layer feedforward neural network), so that an ELM (extreme learning machine) model is conveniently constructed and the solution is accelerated.
Further, the step S202 specifically includes the following steps:
s2021, obtaining dense optical flow, optical flow weight and GRU convolution module output through a GRU convolution module according to the continuous frame RGB image;
s2022, obtaining a camera matrix by using Schur Schur complement of the depth arrowhead-shaped sparse Hessian matrix according to the dense optical flow and the optical flow weight;
s2023, obtaining a first posture according to Cholesky Gerrisby decomposition of the camera matrix;
s2024, obtaining depth according to the first posture;
s2025, carrying out block division on the Hessian matrix to obtain a block division result; the expression of the block segmentation result is:
Figure SMS_1
wherein ,
Figure SMS_2
is a Hessian matrix; c is a block camera matrix;Ea Hessian matrix that is non-diagonal to the camera or depth;
Figure SMS_3
transpose of the Hessian matrix for camera/depth non-diagonal;Pa diagonal matrix corresponding to the depth of each pixel of the key frame;
Figure SMS_4
incremental updates on lie algebra for camera poses in SE (3) group;
Figure SMS_5
an incremental update to the inverse depth per pixel;
Figure SMS_6
is the attitude residual error;wis the depth residual error;bis a residual vector;
s2026, obtaining a depth marginal covariance and an attitude marginal covariance according to the block segmentation result; the expressions of the depth margin covariance and the attitude margin covariance are as follows:
Figure SMS_7
Figure SMS_8
wherein ,
Figure SMS_9
is the depth margin covariance;
Figure SMS_10
the inverse of the diagonal matrix corresponding to the inverse depth of each key frame pixel;
Figure SMS_11
the inverse transpose of the diagonal matrix corresponding to the inverse depth of each key frame pixel;
Figure SMS_12
is attitude margin covariance;Lcholesky factorization for the lower triangular matrix;
Figure SMS_13
transpose of Cholesky factorization of lower triangular matrix;
s2027, obtaining an improved NeRF-SLAM model according to the first posture, the depth, the posture marginal covariance, the depth marginal covariance and the continuous frame RGB image.
The beneficial effects of the above further scheme are: and optimizing the NeRF-SLAM model by using the structure of the Hessian matrix to obtain an improved NeRF-SLAM model, and preparing for obtaining a drawing result subsequently.
Further, the step S203 specifically includes the following steps:
s2031, acquiring the posture of the camera through an IMU inertial navigation module, and obtaining a second posture by using a posture resolving subunit of a NeRF-SLAM radiation nerve field unit;
s2032, according to the first posture, the depth, the posture marginal covariance, the depth marginal covariance and the second posture, high-frequency coding is carried out through a calculation and position coding module to obtain a scene position and an observation direction; the expression of the high-frequency coding is as follows:
Figure SMS_14
wherein ,
Figure SMS_15
is highA function of frequency encoding;pis a position vector;
Figure SMS_16
is a sine function;
Figure SMS_17
is a cosine function;
s2033, obtaining voxel density and feature vectors by using a first single hidden layer feedforward neural network SLFNN1 according to the scene position;
s2034, obtaining color by using a second single hidden layer feedforward neural network SLFNN2 according to the observation direction and the characteristic vector;
s2035, obtaining voxel output through a volume rendering module according to the voxel density and color; the voxel output expression is:
Figure SMS_18
wherein ,
Figure SMS_20
outputting for the voxel;
Figure SMS_24
is a ray from
Figure SMS_26
TotCumulative transmission along the ray, i.e. ray
Figure SMS_21
TotProbability of not encountering any particle;
Figure SMS_22
as a function of voxel density;
Figure SMS_25
is a function of color;
Figure SMS_27
the corresponding light ray;dis a direction vector;
Figure SMS_19
is a proximal boundary;
Figure SMS_23
is a distal boundary;tthe distance from the sampling point on the camera light to the optical center of the camera;sis the position on the light;
s2036, combining the voxel output and the GRU convolution module output, and obtaining a mapping result by utilizing a NeRF-SLAM mapping subunit of the NeRF-SLAM radiation nerve field unit through local mapping, obstacle area judgment and grid map occupation.
The beneficial effects of the above further scheme are: and obtaining a mapping result according to the improved NeRF-SLAM model, and preparing for flight path planning.
Further, the loss function of the mapping result is:
Figure SMS_28
Figure SMS_29
Figure SMS_30
wherein ,
Figure SMS_32
a loss function for the charting result;Tis in a first posture;
Figure SMS_35
is a neural parameter;
Figure SMS_37
is a color loss function;Ithe color image to be rendered;
Figure SMS_33
is a rendered color image;
Figure SMS_34
is a hyper-parameter for balancing depth and color supervision;
Figure SMS_36
is a depth loss function;Dis depth;
Figure SMS_38
is uncertainty margin covariance;
Figure SMS_31
is the rendered depth.
The beneficial effects of the above further scheme are: parameters are continuously adjusted according to the loss function of the mapping result, and the reliability of autonomous flight of the unmanned aerial vehicle is improved.
Drawings
FIG. 1 is a system block diagram of the present invention.
FIG. 2 is a flow chart of the method of the present invention.
Fig. 3 is a schematic structural diagram of a basic frame of the SLFNN hardware unit design in embodiment 3 of the present invention.
Fig. 4 is a flowchart of training the first single hidden layer feedforward neural network or the second single hidden layer feedforward neural network in embodiment 3 of the present invention.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined by the appended claims, and all changes that can be made by the invention using the inventive concept are intended to be protected.
Example 1
As shown in fig. 1, a heterogeneous unmanned aerial vehicle visual obstacle avoidance system based on a NeRF neural network comprises a core module, and a laser radar module, an unmanned aerial vehicle flight control module, an unmanned aerial vehicle monocular camera module and an IMU inertial navigation module which are respectively connected with the core module;
the unmanned aerial vehicle monocular camera module is used for collecting images to obtain original images;
the IMU inertial navigation module is used for acquiring the camera attitude;
the core module is used for obtaining a global map according to the original image and the camera posture;
the laser radar module is used for obtaining a path planning result by combining the global map and the core module;
and the unmanned aerial vehicle flight control module is used for controlling the unmanned aerial vehicle according to the path planning result to realize obstacle avoidance of the unmanned aerial vehicle.
The core module comprises an APU application processor submodule and an FPGA editable logic submodule connected with the APU application processor submodule;
the APU application processor sub-module comprises a laser radar SLAM unit, a global map unit and a flight path planning unit which are sequentially connected; the laser radar SLAM unit is connected with the laser radar module; the global map unit is connected with the FPGA editable logic sub-module; the flight path planning unit is connected with the unmanned aerial vehicle flight control module;
the FPGA editable logic sub-module comprises a MicroBlaze processor unit and a NeRF-SLAM radiation nerve field unit which are sequentially connected; the NeRF-SLAM radiation nerve field unit comprises a NeRF-SLAM mapping subunit, an image acquisition subunit and an attitude calculation subunit which are respectively connected with the NeRF-SLAM mapping subunit; the MicroBlaze processor unit is respectively connected with the NeRF-SLAM map building subunit and the global map unit; the image acquisition subunit is connected with the monocular camera module of the unmanned aerial vehicle; and the attitude resolving subunit is connected with the IMU inertial navigation module.
In this embodiment, the core module adopts a development board with a model of Zynq uidrale + MPSoCXCZU5 EV.
The working principle of the invention is as follows: according to the invention, an unmanned aerial vehicle monocular camera module is used for collecting images, an image collecting subunit is arranged in an FPGA editable logic submodule of a core module, the collected images are transmitted to a NeRF-SLAM mapping subunit, and the NeRF-SLAM mapping subunit is combined with an IMU inertial navigation module and an attitude resolving subunit, so that a scene is modeled into a continuous 5D radiation field, and SLAM mapping can be implicitly carried out through a three-dimensional scene stored in a neural network. The map building result is uploaded to an APU application processor submodule through a MicroBlaze processor unit to update a global map of an obstacle avoidance system, planning of a flight path is achieved by combining a laser radar module and a laser radar SLAM unit, the path planning result is sent to an unmanned aerial vehicle flight control system to control the unmanned aerial vehicle, and visual obstacle avoidance of the unmanned aerial vehicle is achieved.
Example 2
As shown in fig. 2, the invention provides a NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance method, which includes the following steps:
s1, obtaining an original image through a monocular camera module of an unmanned aerial vehicle;
s2, modeling the original image into a continuous 5D radiation field by using a NeRF-SLAM radiation nerve field unit to obtain a mapping result;
s3, transmitting the mapping result to a global map unit through a MicroBlaze processor unit to obtain a global map;
s4, obtaining a path planning result by utilizing a laser radar SLAM unit and a laser radar module according to the global map;
and S5, transmitting the path planning result to an unmanned aerial vehicle flight control module by using a flight path planning unit, and realizing the visual obstacle avoidance of the unmanned aerial vehicle.
The step S2 is specifically as follows:
s201, according to the original image, utilizing a NeRF-SLAM to radiate an image acquisition subunit of a neural field unit to obtain a continuous frame RGB image;
s202, optimizing a NeRF-SLAM model by utilizing the structure of a Hessian matrix according to the continuous frame RGB images to obtain an improved NeRF-SLAM model;
s203, obtaining a mapping result according to the improved NeRF-SLAM model.
The improved NeRF-SLAM model in the step S202 comprises a first single hidden layer feedforward neural network SLFNN1, a second single hidden layer feedforward neural network SLFNN2, a calculation and position coding module, a volume rendering module and a GRU convolution module;
the calculation and position coding module is used for acquiring a scene position and an observation direction;
the first single hidden layer feedforward neural network SLFNN1 is used for obtaining voxel density and feature vectors according to the scene position;
the second single hidden layer feedforward neural network SLFNN2 is used for obtaining colors according to the observation direction and the characteristic vector;
the volume rendering module is used for obtaining voxel output according to the voxel density and the color;
and the GRU convolution module is used for acquiring the output of the GRU convolution module.
The step S202 is specifically as follows:
s2021, obtaining dense optical flow, optical flow weight and GRU convolution module output through a GRU convolution module according to the continuous frame RGB image;
s2022, obtaining a camera matrix by using Schur Schur complement of the depth arrowhead-shaped sparse Hessian matrix according to the dense optical flow and the optical flow weight;
s2023, obtaining a first posture according to Cholesky Gerrisby decomposition of the camera matrix;
s2024, obtaining depth according to the first posture;
s2025, carrying out block division on the Hessian matrix to obtain a block division result; the expression of the block segmentation result is:
Figure SMS_39
wherein ,
Figure SMS_40
is a Hessian matrix; c is a block camera matrix;Ea Hessian matrix that is non-diagonal to the camera or depth;
Figure SMS_41
transpose of the Hessian matrix for camera/depth non-diagonal;Pa diagonal matrix corresponding to the depth of each pixel of the key frame;
Figure SMS_42
incremental updates on lie algebra for camera poses in SE (3) group;
Figure SMS_43
an incremental update to the inverse depth per pixel;
Figure SMS_44
is the attitude residual error;wis the depth residual error;bis a residual vector;
s2026, obtaining a depth marginal covariance and an attitude marginal covariance according to the block segmentation result; the expressions of the depth margin covariance and the attitude margin covariance are as follows:
Figure SMS_45
Figure SMS_46
wherein ,
Figure SMS_47
is the depth margin covariance;
Figure SMS_48
the inverse of the diagonal matrix corresponding to the inverse depth of each key frame pixel;
Figure SMS_49
the inverse transpose of the diagonal matrix corresponding to the inverse depth of each key frame pixel;
Figure SMS_50
is attitude margin covariance;Lcholesky factorization for the lower triangular matrix;
Figure SMS_51
transpose of Cholesky factorization of lower triangular matrix;
s2027, obtaining an improved NeRF-SLAM model according to the first posture, the depth, the posture marginal covariance, the depth marginal covariance and the continuous frame RGB image.
The step S203 is specifically as follows:
s2031, acquiring the posture of the camera through an IMU inertial navigation module, and obtaining a second posture by using a posture resolving subunit of a NeRF-SLAM radiation nerve field unit;
s2032, according to the first posture, the depth, the posture marginal covariance, the depth marginal covariance and the second posture, high-frequency coding is carried out through a calculation and position coding module to obtain a scene position and an observation direction; the expression of the high-frequency coding is as follows:
Figure SMS_52
wherein ,
Figure SMS_53
as a function of the high frequency encoding;pis a position vector;
Figure SMS_54
is a sine function;
Figure SMS_55
is a cosine function;
s2033, obtaining voxel density and feature vectors by using a first single hidden layer feedforward neural network SLFNN1 according to the scene position;
s2034, obtaining color by using a second single hidden layer feedforward neural network SLFNN2 according to the observation direction and the characteristic vector;
s2035, obtaining voxel output through a volume rendering module according to the voxel density and color; the expression of the voxel output is:
Figure SMS_56
wherein ,
Figure SMS_58
outputting for the voxel;
Figure SMS_61
is a ray from
Figure SMS_63
TotCumulative transmission along the ray, i.e. ray
Figure SMS_59
TotProbability of not encountering any particle;
Figure SMS_62
as a function of voxel density;
Figure SMS_64
is a function of color;
Figure SMS_65
corresponding light rays;dis a direction vector;
Figure SMS_57
is a proximal boundary;
Figure SMS_60
is a distal boundary;tthe distance from the sampling point on the camera light to the optical center of the camera;sis the position on the light;
s2036, combining the voxel output and the GRU convolution module output, and obtaining a mapping result by using a NeRF-SLAM mapping subunit of a NeRF-SLAM radiation nerve field unit through local mapping, obstacle region judgment and grid map occupation.
The loss function of the mapping result is:
Figure SMS_66
Figure SMS_67
Figure SMS_68
wherein ,
Figure SMS_71
a loss function for the mapping result;Tis in a first posture;
Figure SMS_72
is a neural parameter;
Figure SMS_74
is a color loss function;Ithe color image to be rendered;
Figure SMS_70
is a rendered color image;
Figure SMS_73
is a hyper-parameter for balancing depth and color supervision;
Figure SMS_75
is a depth loss function;Dis depth;
Figure SMS_76
is uncertainty margin covariance;
Figure SMS_69
is the rendered depth.
Example 3
As shown in figure 1, the heterogeneous unmanned aerial vehicle visual obstacle avoidance system based on the NeRF neural network mainly comprises a core module, a laser radar module, an unmanned aerial vehicle flight control module, an unmanned aerial vehicle monocular camera module and an IMU inertial navigation module. The core module mainly comprises an APU application program processor submodule and an FPGA editable logic submodule; the core module adopts the model to be Zynq UItrascale + MPSoCXCZU5 EV's development board. The APU application processor sub-module is mainly composed of a laser radar SLAM unit, a global map unit and a flight path planning unit. The FPGA editable logic sub-module mainly comprises a MicroBlaze processor unit and a NeRF-SLAM radiation nerve field unit; the NeRF-SLAM radiation nerve field unit comprises a NeRF-SLAM mapping subunit, an image acquisition subunit and an attitude calculation subunit. The method adopts a heterogeneous processing thought, wherein an FPGA editable logic sub-module replaces an MLP multi-layer perceptron in the NeRF-SLAM model, an ELM solves a first single-hidden-layer feedforward neural network SLFNN1 and a second single-hidden-layer feedforward neural network SLFNN2 by adopting a heterogeneous FPGA editable logic sub-module resource acceleration limit learning machine, improves the generalization capability of a nerve radiation field of the NeRF-SLAM model, cannot fall into a local minimum value, and completes drawing construction through a heterogeneous APU application program processor sub-module. In addition, the monocular camera module of the unmanned aerial vehicle is used as an unmanned aerial vehicle visual obstacle avoidance SLAM mapping sensor, monocular depth estimation is carried out by utilizing an intensive light stream estimation algorithm, the weight and the size of the unmanned aerial vehicle are reduced, the cost is reduced, and the reliability of autonomous flight of the unmanned aerial vehicle is improved.
In this embodiment, the MicroBlaze processor unit adopted by the improved NeRF-SLAM model is based on RISC (reduced instruction set processor), which has the advantages of high speed, reduced design cost, and improved reliability.
The FPGA editable logic sub-module mainly comprises a MicroBlaze processor unit and a NeRF-SLAM radiation nerve field unit. The MicroBlaze processor unit is mainly used for managing the NeRF-SLAM radiation nerve field unit, finishing communication with a general purpose input/output (GPIO) interface and a high-speed peripheral I/O interface and processing a clock; the functions comprise (1) system control, (2) data preprocessing, (3) ELM training and (4) map building control.
Further, the MicroBlaze processor unit is connected with a BRAM through an LMB local memory bus, and the storage neuron coefficient can be flexibly changed. Furthermore, considering that implementation of ELM requires a large amount of on-chip memory resources, the use of BRAM is also advantageous to avoid delays caused by off-chip accesses.
Further, the MicroBlaze processor unit is connected to an EMC external storage controller via an XCL cache link to store data into a cache, such as an SD card.
Further, the MicroBlaze processor unit accesses a general purpose I/O interface (GPIO), a high speed peripheral I/O interface and a clock through a PLB peripheral local bus.
Further, after the MicroBlaze processor unit completes data processing work through the FPU floating point processor, the data processing result is transmitted to the PS through the AXI bus.
Furthermore, the NeRF-SLAM radiation nerve field unit acquires original images through an external unmanned aerial vehicle monocular camera module, and continuous frame image input is obtained through an image acquisition subunitI(ii) a Starting with a series of images img 1 to img n, the modified NeRF-SLAM model first computes the dense optical flow between pairs of frames, using the GRU convolution module, given the correlation between pairs of frames and a guess of the current dense optical flow, computes a new dense optical flow, and a weight for each optical flow measurement.
Further, using these streams and weights as metric values, the improved NeRF-SLAM model employed by the present invention solves a dense Beam Adjustment (BA) problem where the 3D geometry is parameterized as a set of inverse depth maps for each keyframe. Parameterization of this structure results in a very efficient approach to solving the dense BA problem, which can be achieved by linearizing the equation set into a familiar camera or depth arrow-like block sparse Hessian matrix
Figure SMS_77
Is expressed as a linear least squares problem, whereincAndpis the dimension of the camera and the point.
Further, to solve the linear least squares problem, the Schur complement of the Hessian matrix is used to compute the simplified camera matrix, since it is not depth dependent and the dimensions are much smaller.
Further, the problem of the camera pose is solved through Cholesky factorization, and the pose is solved through front replacement and back replacementT. Given these posesTWe can solve for the sample depth. Further, given a gestureTAnd depthDThe modified NeRF-SLAM model proposes to compute induced optical flow and provide it again to the GRU convolution module as an initial guess.
Further, the improved NeRF-SLAM model needs to be optimized by using a structure of a Hessian matrix, and block division is performed as follows:
Figure SMS_78
；
Figure SMS_79
；
wherein ,His a Hessian matrix, and the Hessian matrix,bis a residual error that is a function of the error,Cis a matrix of block cameras that is,Pis a diagonal matrix corresponding to the depth of each pixel of the key frame. By using
Figure SMS_80
Incremental updates on lie algebra representing camera pose in SE (3) group, and
Figure SMS_81
is an incremental update to the inverse depth per pixel.EIs a Hessian matrix of cameras or depths that are non-diagonal,
Figure SMS_82
andwcorresponding to pose and depth residuals.
Further, from block segmentation of the Hessian matrix, the dense depth can be efficiently calculated
Figure SMS_83
And posture
Figure SMS_84
Marginal covariance of (2):
Figure SMS_85
Figure SMS_86
wherein ,
Figure SMS_87
is the inverse of the diagonal matrix corresponding to the inverse depth of each key frame pixel,
Figure SMS_88
is the inverse transpose of the diagonal matrix corresponding to the inverse depth of each key frame pixel.EIs a Hessian block matrix of cameras or depth non-diagonals,
Figure SMS_89
is the transpose of the block matrix of the camera/depth non-diagonal Hessian,Lis the Cholesky factorization of the lower triangular matrix,
Figure SMS_90
is the transpose of the Cholesky factorization of the lower triangular matrix,
Figure SMS_91
representing an inverse matrix.
Further, given all the information computed by the tracking module-pose, depth, their respective marginal covariances, and the input RGB image-the radiation field parameters can be optimized and the camera pose refined at the same time.
Furthermore, the MLP multilayer perceptron in the classical NeRF network is replaced by a first single-hidden-layer feedforward neural network SLFNN1 and a second single-hidden-layer feedforward neural network SLFNN2 which are realized based on FPGA editable logic sub-modules, so that an ELM (extreme learning machine) model can be conveniently constructed and the solution can be accelerated.
Further, the IMU inertial navigation module acquires the camera attitude, a second attitude Q is obtained through an attitude resolving subunit, and the second attitude Q is sent to the calculation and position coding module, and the depth covariance generated in the front is sent to the calculation and position coding module
Figure SMS_92
Combined depthDAttitude covariance
Figure SMS_93
Unite the first postureTAnd are also connected to the calculation and position coding module. Calculating scene position (x, y, z) and viewing direction of points in a scene
Figure SMS_94
Wherein the scene position (x, y, z) is encodedThen sending the scene position into a first single hidden layer feedforward neural network SLFNN1, after passing through (1) a random neuron, (2) an activation function lookup table and (3) an output neuron, outputting the voxel density through a maximum output selection module of the first single hidden layer feedforward neural network SLFNN1 and outputting the voxel density
Figure SMS_95
And 256-dimensional feature vectors, and then the 256-dimensional feature vectors obtained above are associated with the viewing direction
Figure SMS_96
After Concat combination, processing by a second single hidden layer feedforward neural network SLFNN2, passing through (1) a random neuron, (2) an activation function lookup table and (3) an output neuron, and then outputting color RGB through a maximum output selection module of the second single hidden layer feedforward neural network SLFNN2.
Further, the MicroBlaze processor unit trains a first single hidden layer feedforward neural network SLFNN1 and a second single hidden layer feedforward neural network SLFNN2 through an FPGA editable logic sub-module fast unidirectional link FSL. The first single hidden layer feedforward neural network SLFNN1 and the second single hidden layer feedforward neural network SLFNN2 are composed of (1) random neurons, (2) an activation function lookup table, (3) output neurons and a maximum output selection module.
Further, in the calculation and position coding module, improving the NeRF-SLAM model requires high-frequency details of the scene to reconstruct a high-definition scene. The high-frequency coding function adopted in the coding process is as follows:
Figure SMS_97
wherein ,pin the form of a position vector, the position vector,
Figure SMS_98
the representation is taken as a sine wave,
Figure SMS_99
the representation is taken as the cosine of the,πtake 3.1415926.
The first single hidden layer feedforward neural network SLFNN1 and the second single hidden layer feedforward neural network SLFNN2 are common processing methods combined with FPGA editable logic sub-modules such as parallel and assembly lines, a basic frame of hardware unit design can be obtained, as shown in FIG. 3, in the execution process of the ELM algorithm, a series of numerical values, such as random weights, need to be generated
Figure SMS_100
And bias
Figure SMS_101
And solving for the weights
Figure SMS_102
. M +1 input data realize specific digital coding by random weight from the input module, are input into the neuron module through the multiplexer, and output calculation results outwards in time. The input module and the multiplexer adopt parallel processing, and the neuron module and the output module adopt serial processing.
Further, the voxel density
Figure SMS_103
And the color RGB, i.e. c = (r, g, b), is connected to the volume rendering module, and the rendered voxel output is obtained by the volume rendering module:
Figure SMS_104
wherein ,
Figure SMS_105
representing voxel output, function
Figure SMS_106
Represents rays from
Figure SMS_107
TotCumulative transmission along the ray, i.e. ray from
Figure SMS_108
TotDo not touchThe probability of any particle being present,
Figure SMS_109
as a function of voxel density;
Figure SMS_110
is a function of color;
Figure SMS_111
the corresponding light ray;dis a direction vector.
In the HLS high-level synthesis adopted in the actual FPGA editable logic sub-module, the method is realized in a discretization mode:
Figure SMS_112
wherein ,
Figure SMS_113
；
Figure SMS_114
；
wherein
Figure SMS_115
Is output for the discretized voxels,
Figure SMS_116
is a discretized transmission.
In this embodiment, HLS high-level synthesis refers to a process of automatically converting a logic structure described in a high-level language into a circuit model described in a low-abstraction-level language.
Further, the calculation result of the voxel calculation and the output of the former GRU convolution module are Concat and are output to a NeRF-SLAM mapping subunit for mapping. And after the NeRF-SLAM mapping subunit performs three steps of local mapping, obstacle area judgment and grid map occupation, the mapping result is output to the MicroBlaze processor unit through the fast unidirectional link FSL of the FPGA editable logic submodule.
Further, considering uncertainty perception loss, the mapping loss function is expressed as:
Figure SMS_117
given hyper-parameter
Figure SMS_118
Depth of balance, we are right to attitudeTAnd neural parameters
Figure SMS_119
All minimize and color monitor (will)
Figure SMS_120
Set to 1.0).
Further, the depth loss function may be expressed as:
Figure SMS_121
wherein
Figure SMS_122
Is the depth of rendering, andDdense depth and uncertainty estimated for the tracking module. We render depth as the expected ray termination distance.
Further, the depth of each pixel is evaluated by sampling the three-dimensional location along the ray of the pixel, evaluating the sampleiDensity of (2)
Figure SMS_123
And alpha synthesizing the calculated result density, similar to standard volume rendering; the expression for pixel depth is:
Figure SMS_124
according to the sampleiAlong the depth of the ray or rays,
Figure SMS_125
is the distance between successive samples and is used as an input variable for pixel depthAmount of the compound (A). Wherein
Figure SMS_126
Is the bulk density of the sampleiIs generated by evaluating an MLP on the three-dimensional world coordinates.
Further, the air conditioner is provided with a fan,
Figure SMS_127
is along the light to the sampleiIs defined as:
Figure SMS_128
further, the color loss function is defined as, in the original NeRF:
Figure SMS_129
wherein
Figure SMS_130
Is a rendered color image, synthesized similar to a depth image, by using volume rendering. Each color of each pixel is also calculated by sampling the ray along the pixel and alpha synthesizing the resulting density and color:
Figure SMS_131
, wherein
Figure SMS_132
Is also that
Figure SMS_133
And a transmittance of
Figure SMS_134
Is a color estimated by an ELM (extreme learning machine). For a given sampleiWhile estimating the density
Figure SMS_135
And color
Figure SMS_136
。
Further, the training process of the first single hidden layer feedforward neural network SLFNN1 and the second single hidden layer feedforward neural network SLFNN2 adopted in the improved NeRF-SLAM model is shown in FIG. 4. The method mainly comprises the steps of designing an external software unit and an internal hardware unit, and firstly, starting a training process in the first step; secondly, inputting preprocessing data; thirdly, processing the training labels; fourthly, entering a hardware unit design and distributing random weight; fifthly, calculating hidden layer output; sixthly, calculating the generalized inverse of the matrix; seventhly, calculating network output; eighthly, outputting data accuracy; and step nine, judging whether the training is needed again, if so, re-executing the steps four, five, six, seven and eight, and if not, ending the training process.
In conclusion, the invention provides that the unmanned aerial vehicle acquires images through the monocular camera module of the unmanned aerial vehicle in the autonomous flight process, the FPGA editable logic sub-module of the heterogeneous processor core module is internally provided with the image acquisition sub-unit, and the acquired images are transmitted to the NeRF-SLAM image establishing sub-unit; the neural network-based SLAM mapping method based on the IMU inertial navigation module is characterized by combining the IMU inertial navigation module and the attitude calculation subunit, the NeRF-SLAM mapping subunit models a scene into a continuous 5D radiation field, and the SLAM mapping can be implicitly performed through a three-dimensional scene stored in a neural network. The map building result is uploaded to an APU application processor submodule through a MicroBlaze processor unit based on RISC (reduced instruction set processor) to update a global map of an obstacle avoidance system, a flight path is planned by combining a laser radar module and a laser radar SLAM unit, and the path planning result is sent to an unmanned aerial vehicle flight control module to control the unmanned aerial vehicle.

Claims (8)

1. A heterogeneous unmanned aerial vehicle visual obstacle avoidance system based on a NeRF neural network is characterized by comprising a core module, a laser radar module, an unmanned aerial vehicle flight control module, an unmanned aerial vehicle monocular camera module and an IMU inertial navigation module, wherein the laser radar module, the unmanned aerial vehicle flight control module, the unmanned aerial vehicle monocular camera module and the IMU inertial navigation module are respectively connected with the core module;
the unmanned aerial vehicle monocular camera module is used for collecting images to obtain original images;
the IMU inertial navigation module is used for acquiring the camera attitude;
the core module is used for obtaining a global map according to the original image and the camera posture;
the laser radar module is used for combining the core module according to the global map to obtain a path planning result;
and the unmanned aerial vehicle flight control module is used for controlling the unmanned aerial vehicle according to the path planning result to realize obstacle avoidance of the unmanned aerial vehicle.
2. The NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system according to claim 1, wherein the core module comprises an APU application processor submodule and an FPGA editable logic submodule connected with the APU application processor submodule;
the APU application processor sub-module comprises a laser radar SLAM unit, a global map unit and a flight path planning unit which are sequentially connected; the laser radar SLAM unit is connected with the laser radar module; the global map unit is connected with the FPGA editable logic sub-module; the flight path planning unit is connected with the unmanned aerial vehicle flight control module;
the FPGA editable logic sub-module comprises a MicroBlaze processor unit and a NeRF-SLAM radiation nerve field unit which are sequentially connected; the NeRF-SLAM radiation nerve field unit comprises a NeRF-SLAM mapping subunit, an image acquisition subunit and an attitude calculation subunit which are respectively connected with the NeRF-SLAM mapping subunit; the MicroBlaze processor unit is respectively connected with the NeRF-SLAM map building subunit and the global map unit; the image acquisition subunit is connected with the monocular camera module of the unmanned aerial vehicle; and the attitude resolving subunit is connected with the IMU inertial navigation module.
3. The obstacle avoidance method of the NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system, as claimed in any one of claims 1-2, wherein the NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance method comprises the following steps:
s1, obtaining an original image through a monocular camera module of an unmanned aerial vehicle;
s2, modeling the original image into a continuous 5D radiation field by using a NeRF-SLAM radiation nerve field unit to obtain a mapping result;
s3, transmitting the mapping result to a global map unit through a MicroBlaze processor unit to obtain a global map;
s4, obtaining a path planning result by utilizing a laser radar SLAM unit and a laser radar module according to the global map;
and S5, transmitting the path planning result to an unmanned aerial vehicle flight control module by using a flight path planning unit, and realizing the visual obstacle avoidance of the unmanned aerial vehicle.
4. The obstacle avoidance method of the NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system, as claimed in claim 3, wherein the step S2 is as follows:
s201, according to the original image, utilizing a NeRF-SLAM to radiate an image acquisition subunit of a neural field unit to obtain a continuous frame RGB image;
s202, optimizing a NeRF-SLAM model by utilizing the structure of a Hessian matrix according to the continuous frame RGB images to obtain an improved NeRF-SLAM model;
s203, obtaining a mapping result according to the improved NeRF-SLAM model.
5. The obstacle avoidance method of the NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system, as claimed in claim 4, wherein the improved NeRF-SLAM model in step S202 comprises a first single hidden layer feed-forward neural network SLFNN1, a second single hidden layer feed-forward neural network SLFNN2, a calculation and position coding module, a volume rendering module and a GRU convolution module;
the calculation and position coding module is used for acquiring a scene position and an observation direction;
the first single hidden layer feedforward neural network SLFNN1 is used for obtaining voxel density and feature vectors according to the scene position;
the second single hidden layer feedforward neural network SLFNN2 is used for obtaining colors according to the observation direction and the characteristic vector;
the volume rendering module is used for obtaining voxel output according to the voxel density and the color;
and the GRU convolution module is used for acquiring the output of the GRU convolution module.
6. The obstacle avoidance method of the NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system, as claimed in claim 5, wherein the step S202 is as follows:
s2021, obtaining dense optical flow, optical flow weight and GRU convolution module output through a GRU convolution module according to the continuous frame RGB image;
s2022, obtaining a camera matrix by using Schur Schur complement of the depth arrowhead-shaped sparse Hessian matrix according to the dense optical flow and the optical flow weight;
s2023, obtaining a first posture according to Cholesky Gerrisby decomposition of the camera matrix;
s2024, obtaining depth according to the first posture;
s2025, carrying out block division on the Hessian matrix to obtain a block division result; the expression of the block segmentation result is:
Figure QLYQS_1
wherein ,
Figure QLYQS_2
is a Hessian matrix; c is a block camera matrix;Ea Hessian matrix that is non-diagonal to the camera or depth;
Figure QLYQS_3
transpose of the Hessian matrix for camera/depth non-diagonal;Pa diagonal matrix corresponding to the depth of each pixel of the key frame;
Figure QLYQS_4
incremental updates on lie algebra for camera poses in SE (3) group;
Figure QLYQS_5
an incremental update to the inverse depth per pixel;
Figure QLYQS_6
is the attitude residual error;wis the depth residual error;bis a residual vector;
s2026, obtaining a depth marginal covariance and an attitude marginal covariance according to the block segmentation result; the expressions of the depth margin covariance and the attitude margin covariance are as follows:
Figure QLYQS_7
Figure QLYQS_8
wherein ,
Figure QLYQS_9
is the depth margin covariance;
Figure QLYQS_10
the inverse of the diagonal matrix corresponding to the inverse depth of each key frame pixel;
Figure QLYQS_11
the inverse transpose of the diagonal matrix corresponding to the inverse depth of each key frame pixel;
Figure QLYQS_12
is attitude margin covariance;Lcholesky factorization for the lower triangular matrix;
Figure QLYQS_13
cholesky Cholesy for lower triangular matricesTransposing a base decomposition factor;
s2027, obtaining an improved NeRF-SLAM model according to the first posture, the depth, the posture marginal covariance, the depth marginal covariance and the continuous frame RGB image.
7. The obstacle avoidance method of the NeRF neural network-based heterogeneous unmanned aerial vehicle visual obstacle avoidance system according to claim 6, wherein the step S203 is specifically as follows:
s2031, acquiring the posture of the camera through an IMU inertial navigation module, and obtaining a second posture by using a posture resolving subunit of a NeRF-SLAM radiation nerve field unit;
s2032, according to the first posture, the depth, the posture marginal covariance, the depth marginal covariance and the second posture, high-frequency coding is carried out through a calculation and position coding module to obtain a scene position and an observation direction; the expression of the high-frequency coding is as follows:
Figure QLYQS_14
wherein ,
Figure QLYQS_15
as a function of the high frequency encoding;pis a position vector;
Figure QLYQS_16
is a sine function;
Figure QLYQS_17
is a cosine function;
s2033, obtaining voxel density and feature vectors by using a first single hidden layer feedforward neural network SLFNN1 according to the scene position;
s2034, obtaining the color by utilizing a second single hidden layer feedforward neural network SLFNN2 according to the observation direction and the characteristic vector;
s2035, obtaining voxel output through a volume rendering module according to the voxel density and color; the voxel output expression is:
Figure QLYQS_18
wherein ,
Figure QLYQS_21
outputting for the voxel;
Figure QLYQS_23
is a ray from
Figure QLYQS_25
TotCumulative transmission along the ray, i.e. ray
Figure QLYQS_20
TotProbability of not encountering any particle;
Figure QLYQS_24
as a function of voxel density;
Figure QLYQS_26
is a function of color;
Figure QLYQS_27
the corresponding light ray;dis a direction vector;
Figure QLYQS_19
is a proximal boundary;
Figure QLYQS_22
is a distal boundary;tthe distance from the sampling point on the camera light to the optical center of the camera;sis the position on the light;
s2036, combining the voxel output and the GRU convolution module output, and obtaining a mapping result by using a NeRF-SLAM mapping subunit of a NeRF-SLAM radiation nerve field unit through local mapping, obstacle region judgment and grid map occupation.
8. The obstacle avoidance method of the heterogeneous unmanned aerial vehicle visual obstacle avoidance system based on the NeRF neural network, as recited in claim 7, wherein the loss function of the mapping result is:
Figure QLYQS_28
Figure QLYQS_29
Figure QLYQS_30
wherein ,
Figure QLYQS_32
a loss function for the charting result;Tis in a first posture;
Figure QLYQS_35
is a neural parameter;
Figure QLYQS_37
is a color loss function;Ithe color image to be rendered;
Figure QLYQS_33
is a rendered color image;
Figure QLYQS_34
is a hyper-parameter for balancing depth and color supervision;
Figure QLYQS_36
is a depth loss function;Dis depth;
Figure QLYQS_38
is uncertainty margin covariance;
Figure QLYQS_31
is the rendered depth.
CN202310147930.9A 2023-02-22 2023-02-22 Heterogeneous unmanned aerial vehicle vision obstacle avoidance system and method based on NeRF neural network Active CN115826628B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310147930.9A CN115826628B (en) 2023-02-22 2023-02-22 Heterogeneous unmanned aerial vehicle vision obstacle avoidance system and method based on NeRF neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310147930.9A CN115826628B (en) 2023-02-22 2023-02-22 Heterogeneous unmanned aerial vehicle vision obstacle avoidance system and method based on NeRF neural network

Publications (2)

Publication Number Publication Date
CN115826628A true CN115826628A (en) 2023-03-21
CN115826628B CN115826628B (en) 2023-05-09

Family

ID=85522084

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310147930.9A Active CN115826628B (en) 2023-02-22 2023-02-22 Heterogeneous unmanned aerial vehicle vision obstacle avoidance system and method based on NeRF neural network

Country Status (1)

Country Link
CN (1) CN115826628B (en)

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105059533A (en) * 2015-08-14 2015-11-18 深圳市多翼创新科技有限公司 Aircraft and landing method thereof
CN105759836A (en) * 2016-03-14 2016-07-13 武汉卓拔科技有限公司 Unmanned aerial vehicle obstacle avoidance method and device based on 3D camera
CN106168808A (en) * 2016-08-25 2016-11-30 南京邮电大学 A kind of rotor wing unmanned aerial vehicle automatic cruising method based on degree of depth study and system thereof
US20170301109A1 (en) * 2016-04-15 2017-10-19 Massachusetts Institute Of Technology Systems and methods for dynamic planning and operation of autonomous systems using image observation and information theory
CN107553490A (en) * 2017-09-08 2018-01-09 深圳市唯特视科技有限公司 A kind of monocular vision barrier-avoiding method based on deep learning
CN207157509U (en) * 2017-01-24 2018-03-30 刘畅 A kind of unmanned plane for gunnery training and amusement
CN108875912A (en) * 2018-05-29 2018-11-23 天津科技大学 A kind of neural network model for image recognition
CN110456805A (en) * 2019-06-24 2019-11-15 深圳慈航无人智能系统技术有限公司 A kind of UAV Intelligent tracking flight system and method
CN110873565A (en) * 2019-11-21 2020-03-10 北京航空航天大学 Unmanned aerial vehicle real-time path planning method for urban scene reconstruction
US20200250850A1 (en) * 2019-02-06 2020-08-06 Ford Global Technologies, Llc Hybrid Metric-Topological Camera-Based Localization
CN111831010A (en) * 2020-07-15 2020-10-27 武汉大学 Unmanned aerial vehicle obstacle avoidance flight method based on digital space slice
CN112435325A (en) * 2020-09-29 2021-03-02 北京航空航天大学 VI-SLAM and depth estimation network-based unmanned aerial vehicle scene density reconstruction method
CN113192182A (en) * 2021-04-29 2021-07-30 山东产研信息与人工智能融合研究院有限公司 Multi-sensor-based live-action reconstruction method and system
CN113405547A (en) * 2021-05-21 2021-09-17 杭州电子科技大学 Unmanned aerial vehicle navigation method based on semantic VSLAM
US20210350162A1 (en) * 2020-05-07 2021-11-11 Skydio, Inc. Visual observer for unmanned aerial vehicles
CN114170459A (en) * 2021-11-23 2022-03-11 杭州弥拓科技有限公司 Dynamic building identification guiding method and system based on Internet of things and electronic equipment

Patent Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105059533A (en) * 2015-08-14 2015-11-18 深圳市多翼创新科技有限公司 Aircraft and landing method thereof
CN105759836A (en) * 2016-03-14 2016-07-13 武汉卓拔科技有限公司 Unmanned aerial vehicle obstacle avoidance method and device based on 3D camera
US20170301109A1 (en) * 2016-04-15 2017-10-19 Massachusetts Institute Of Technology Systems and methods for dynamic planning and operation of autonomous systems using image observation and information theory
CN106168808A (en) * 2016-08-25 2016-11-30 南京邮电大学 A kind of rotor wing unmanned aerial vehicle automatic cruising method based on degree of depth study and system thereof
CN207157509U (en) * 2017-01-24 2018-03-30 刘畅 A kind of unmanned plane for gunnery training and amusement
CN107553490A (en) * 2017-09-08 2018-01-09 深圳市唯特视科技有限公司 A kind of monocular vision barrier-avoiding method based on deep learning
CN108875912A (en) * 2018-05-29 2018-11-23 天津科技大学 A kind of neural network model for image recognition
CN111540011A (en) * 2019-02-06 2020-08-14 福特全球技术公司 Hybrid metric-topology camera based positioning
US20200250850A1 (en) * 2019-02-06 2020-08-06 Ford Global Technologies, Llc Hybrid Metric-Topological Camera-Based Localization
CN110456805A (en) * 2019-06-24 2019-11-15 深圳慈航无人智能系统技术有限公司 A kind of UAV Intelligent tracking flight system and method
CN110873565A (en) * 2019-11-21 2020-03-10 北京航空航天大学 Unmanned aerial vehicle real-time path planning method for urban scene reconstruction
US20210158009A1 (en) * 2019-11-21 2021-05-27 Beihang University UAV Real-Time Path Planning Method for Urban Scene Reconstruction
US20210350162A1 (en) * 2020-05-07 2021-11-11 Skydio, Inc. Visual observer for unmanned aerial vehicles
CN111831010A (en) * 2020-07-15 2020-10-27 武汉大学 Unmanned aerial vehicle obstacle avoidance flight method based on digital space slice
CN112435325A (en) * 2020-09-29 2021-03-02 北京航空航天大学 VI-SLAM and depth estimation network-based unmanned aerial vehicle scene density reconstruction method
CN113192182A (en) * 2021-04-29 2021-07-30 山东产研信息与人工智能融合研究院有限公司 Multi-sensor-based live-action reconstruction method and system
CN113405547A (en) * 2021-05-21 2021-09-17 杭州电子科技大学 Unmanned aerial vehicle navigation method based on semantic VSLAM
CN114170459A (en) * 2021-11-23 2022-03-11 杭州弥拓科技有限公司 Dynamic building identification guiding method and system based on Internet of things and electronic equipment

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
CHEN-HSUAN LIN;WEI-CHIU MA;ANTONIO TORRALBA;SIMON LUCEY: "BARF: Bundle-Adjusting Neural Radiance Fields" *
徐风帆: "基于神经辐射场优化的三维建模方法" *
聂欣雨: "基于深度学习的视觉SLAM算法研究" *

Also Published As

Publication number Publication date
CN115826628B (en) 2023-05-09

Similar Documents

Publication Publication Date Title
CN110827415B (en) All-weather unknown environment unmanned autonomous working platform
JP6745328B2 (en) Method and apparatus for recovering point cloud data
Lyu et al. Chipnet: Real-time lidar processing for drivable region segmentation on an fpga
CN111902826A (en) Positioning, mapping and network training
AU2021200832A1 (en) Collaborative 3d mapping and surface registration
CN116645649A (en) Vehicle pose and size estimation method, device and storage medium
Huang et al. Non-model-based monocular pose estimation network for uncooperative spacecraft using convolutional neural network
CN117214904A (en) Intelligent fish identification monitoring method and system based on multi-sensor data
Almanza-Medina et al. Deep learning architectures for navigation using forward looking sonar images
Deng et al. ToF and stereo data fusion using dynamic search range stereo matching
Yao et al. Vision-based environment perception and autonomous obstacle avoidance for unmanned underwater vehicle
JP2023079022A (en) Information processing device and information generation method
CN116152442B (en) Three-dimensional point cloud model generation method and device
Zhu et al. Autonomous reinforcement control of visual underwater vehicles: Real-time experiments using computer vision
CN116452654B (en) BEV perception-based relative pose estimation method, neural network and training method thereof
CN112750155A (en) Panoramic depth estimation method based on convolutional neural network
CN113534189A (en) Weight detection method, human body characteristic parameter detection method and device
CN116486038A (en) Three-dimensional construction network training method, three-dimensional model generation method and device
CN115826628B (en) Heterogeneous unmanned aerial vehicle vision obstacle avoidance system and method based on NeRF neural network
CN116079727A (en) Humanoid robot motion simulation method and device based on 3D human body posture estimation
CN116363205A (en) Space target pose resolving method based on deep learning and computer program product
Heintz et al. Spacecraft State Estimation Using Neural Radiance Fields
CN113808186A (en) Training data generation method and device and electronic equipment
Lee et al. Radar translation network between sunny and rainy domains by combination of KP-convolution and CycleGAN
Zhang et al. A Self-Supervised Monocular Depth Estimation Approach Based on UAV Aerial Images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant