CN113624231A - Inertial vision integrated navigation positioning method based on heterogeneous image matching and aircraft - Google Patents

Abstract

The invention provides an inertial vision integrated navigation positioning method based on heterogeneous image matching and an aircraft, wherein the method comprises the following steps: respectively carrying out orthoimage correction on the real-time image and the reference image, and unifying the scaling sizes of the real-time image and the reference image; respectively acquiring a first projection and a quantized gradient direction histogram of a real-time image and a second projection and a quantized gradient direction histogram of a reference image; detecting the similarity of the first projection and the second projection and the histogram of the quantization gradient direction, wherein the position of the maximum Hamming distance value of the second projection and the histogram of the quantization gradient direction is the image matching position of the real-time image in the reference image; and converting the image matching position into the position of the unmanned aerial vehicle as an observed quantity, and constructing a Kalman filter based on the observed quantity to realize the inertial vision integrated navigation positioning of the unmanned aerial vehicle. By applying the technical scheme of the invention, the technical problem that accurate and stable matching between the heterogeneous images is difficult to realize in the prior art is solved.

Description

Inertial Vision Integrated Navigation and Positioning Method and Aircraft Based on Heterogeneous Image Matching

technical field

The invention relates to the technical field of computer vision research, in particular to an inertial vision combined navigation and positioning method and an aircraft based on heterologous image matching.

Background technique

The reference images used in image matching are visible light satellite images, and real-time images generally use infrared images that can work all day, which is a typical heterogeneous image matching technology. Different from homologous image matching, due to the different imaging mechanisms of heterologous images, the images captured in the same scene will be very different, so special processing is required for heterologous image matching. At the same time, with the change of the flight attitude of the aircraft, the real-time image captured by the camera is not a straight-down view, and there are geometric transformations such as scale, rotation, affine and perspective between the real-time image and the orthophoto reference image preloaded on the aircraft. Therefore, before matching, it is necessary to perform a certain preprocessing correction operation on the image, and at the same time, it is also necessary to take into account the invariance of the designed matching algorithm to a certain degree of image geometric deformation that still exists after correction. However, in the existing visual matching, it is difficult to achieve accurate and stable matching between heterogeneous images, so the combined navigation algorithm is less autonomous.

SUMMARY OF THE INVENTION

The invention provides an inertial vision combined navigation and positioning method and an aircraft based on heterologous image matching, which can solve the technical problem that it is difficult to achieve accurate and stable matching between heterologous images in the prior art.

According to one aspect of the present invention, an inertial vision combined navigation and positioning method based on heterologous image matching is provided. The inertial vision combined navigation and positioning method based on heterologous image matching includes: using inertial navigation attitude information and laser ranging height information to Perform orthophoto correction on the real-time image and the reference image respectively, and unify the scaling size of the real-time image and the baseline image; respectively obtain the first projection and quantized gradient direction histogram of the real-time image and the second projection and quantized gradient direction histogram of the baseline image; Based on the similarity measurement principle, the similarity between the first projection and the quantized gradient direction histogram and the second projection and the quantized gradient direction histogram is detected, and the similarity between the first projection and the quantized gradient direction histogram is retrieved from the second projection and the quantized gradient direction histogram. The position of the maximum Hamming distance of the orientation histogram, the position of the maximum Hamming distance is the image matching position of the real-time image in the reference image; the image matching position is converted to the position of the UAV and used as the observation amount, based on The Kalman filter is constructed from the observations to realize the inertial vision integrated navigation and positioning of the UAV.

Further, obtaining the first projection of the real-time image and the quantized gradient direction histogram specifically includes: performing Gaussian filtering on the real-time image; using the Sobel operator to extract the grayscale image gradient feature of the Gaussian filtered real-time image to obtain the gradient image of the real-time image. ; Calculate the gradient histogram of the real-time graph based on the gradient image of the real-time graph; Project and quantify the gradient histogram of the real-time graph to obtain the first projected and quantized gradient direction histogram; Obtain the second projected and quantized gradient direction of the reference graph The histogram specifically includes: performing Gaussian filtering on the reference image; using the Sobel operator to extract the grayscale image gradient features of the Gaussian filtered baseline image to obtain the gradient image of the baseline image; based on the gradient image of the baseline image, count the gradient histogram of the baseline image. map; project and quantify the gradient histogram of the reference map to obtain a second projected and quantized gradient direction histogram.

实现正射影像校正，其中，

为非正射影像，

为正射影像，

为空间点P在世界坐标系中的齐次坐标，z_c为空间点P在摄像机坐标系中的坐标的z分量，f为光学系统主距，s_c为图像传感器上列方向上相邻像素之间的距离，s_r为图像传感器上行方向上相邻像素之间的距离，[c_c,c_r]^T为图像的主点，

为两个内参完全一致的摄像机之间的旋转矩阵，

为从世界坐标系向摄像机坐标系转换的旋转矩阵，

为从世界坐标系向摄像机坐标系转换的平移向量。Further, both the real-time graph and the benchmark graph can be based on

Implements orthophoto correction, where,

is a non-orthophoto image,

is an orthophoto,

is the homogeneous coordinate of the space point P in the world coordinate system, z _c is the z component of the coordinate of the space point P in the camera coordinate system, f is the principal distance of the optical system, and s _c is the adjacent pixels in the column direction of the image sensor The distance between them, s _r is the distance between adjacent pixels in the upward direction of the image sensor, [c _c ,c _r ] ^T is the main point of the image,

is the rotation matrix between two cameras with identical internal parameters,

is the rotation matrix converted from the world coordinate system to the camera coordinate system,

is the translation vector transformed from the world coordinate system to the camera coordinate system.

来获取，其中，μ为像元尺寸，f为相机焦距，l为相机光心距拍摄点距离。Further, the scaling factor k of the real-time graph and the reference graph can be based on

to obtain, where μ is the pixel size, f is the focal length of the camera, and l is the distance between the optical center of the camera and the shooting point.

来获取，其中，d(a,b)为差别度量，f为单调函数，(u,v)为实时图中的像素坐标集合，I(x+u,y+v)表示实时图在参考图(x,y)处的相似性度量值，T为实时图，I为参考图。Further, the similarity S(x, y) between the first projected and quantized gradient direction histogram and the second projected and quantized gradient direction histogram can be based on

to obtain, where d(a, b) is the difference measure, f is a monotonic function, (u, v) is the set of pixel coordinates in the real-time image, and I(x+u, y+v) indicates that the real-time image is in the reference image The similarity measure at (x, y), where T is the real-time image and I is the reference image.

其中，h为飞机飞行高度，x和y为相机正下方与拍摄图像间位置关系，r^d为相机拍摄位置中心点在相机机体坐标系下的投影，rⁿ为r^d在地理坐标系下投影，

为从相机机体坐标系向地理坐标系转换的矩阵。Further, the conversion relationship between the image matching position and the position of the UAV is as follows:

Among them, h is the flight height of the aircraft, x and y are the positional relationship between the camera directly under the camera and the captured image, r ^d is the projection of the center point of the camera shooting position in the camera body coordinate system, and r ⁿ is the projection of r ^d in the geographic coordinate system ,

is the matrix to convert from the camera body coordinate system to the geographic coordinate system.

Further, in the Kalman filter, the state vector includes north speed error, sky speed error, east speed error, latitude error, altitude error, longitude error, north misalignment angle, sky misalignment angle, and east misalignment angle. , x-axis installation error, y-axis installation error, z-axis installation error and laser ranging scale coefficient error.

r_N为北向距离，r_U为天向距离，r_E为东向距离。Further, in the Kalman filter, the observation matrix is H(k)=[H ₁ H ₂ H ₃ H ₄ H ₅ ], where,

r _N is the distance to the north, r _U is the distance to the sky, and r _E is the distance to the east.

According to another aspect of the present invention, an aircraft is provided, which uses the above-mentioned heterogeneous image matching-based inertial visual integrated navigation and positioning method to perform integrated navigation and positioning.

By applying the technical solution of the present invention, an inertial vision integrated navigation and positioning method based on heterologous image matching is provided. In the research of heterologous image matching and positioning technology, firstly, the method of using laser ranging sensor to capture flight height information is proposed to solve the problem that the scale information cannot be initialized under small maneuvering conditions in the cruise segment. Through the fusion of inertial/laser ranging/image information , to achieve the unification of the scale of the real-time image and the reference image; secondly, the method based on the projection and quantized gradient direction histogram is used to extract and binary code the structural features, so as to solve the problem of mismatching due to inconsistent image information in the process of heterogeneous image matching. A geometric positioning method based on inertial/laser rangefinder is proposed to solve the problem of large attitude error in the process of PNP pose calculation of two-dimensional plane control points during high-altitude flight. This method can realize the difference between heterogeneous images. Stable matching improves the autonomy of the integrated navigation algorithm.

Description of drawings

The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention, constitute a part of the specification, are used to illustrate the embodiments of the invention, and together with the description, serve to explain the principles of the invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 shows a side view of a coordinate system provided according to a specific embodiment of the present invention;

FIG. 2 shows a plan view of a coordinate system provided according to a specific embodiment of the present invention;

FIG. 3 shows a schematic structural diagram of a captured image provided according to a specific embodiment of the present invention;

4 shows a schematic structural diagram of an image correction result provided according to a specific embodiment of the present invention;

5 shows a schematic structural diagram of a comparison of infrared and visible light structural features provided according to a specific embodiment of the present invention;

FIG. 6 shows a flow chart of the projection and quantization gradient direction histogram generation provided according to a specific embodiment of the present invention;

7 shows a schematic structural diagram of a two-dimensional Gaussian function curve provided according to a specific embodiment of the present invention;

FIG. 8 shows a schematic structural diagram of a cell unit provided according to a specific embodiment of the present invention;

FIG. 9 shows a schematic structural diagram of the position of a real-time graph in a reference graph provided according to a specific embodiment of the present invention;

FIG. 10 shows a schematic structural diagram of the positional relationship between the positioning point of the UAV and the image matching point according to a specific embodiment of the present invention.

Detailed ways

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

The relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise. Meanwhile, it should be understood that, for the convenience of description, the dimensions of various parts shown in the accompanying drawings are not drawn in an actual proportional relationship. Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the authorized description. In all examples shown and discussed herein, any specific value should be construed as illustrative only and not as limiting. Accordingly, other examples of exemplary embodiments may have different values. It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

As shown in FIG. 1 to FIG. 10 , according to a specific embodiment of the present invention, an inertial vision combined navigation and positioning method based on heterologous image matching is provided, and the inertial vision combined navigation and positioning method based on heterologous image matching includes: The real-time image and the reference image are respectively corrected by orthophotos based on the orientation information and laser ranging height information, and the scaling size of the real-time image and the baseline image is unified; The second projected and quantized gradient direction histogram; based on the similarity measurement principle, the similarity between the first projected and quantized gradient direction histogram and the second projected and quantized gradient direction histogram is detected, and the second projected and quantized gradient direction histogram Retrieve the position of the maximum Hamming distance from the first projection and the quantized gradient direction histogram in the figure, and the position of the maximum Hamming distance is the image matching position of the real-time image in the reference image; convert the image matching position to no The position of the human and the machine is used as the observation quantity, and the Kalman filter is constructed based on the observation quantity to realize the inertial visual integrated navigation and positioning of the UAV.

Using this configuration method, an inertial vision integrated navigation and positioning method based on heterologous image matching is provided. In the research of source image matching and positioning technology, firstly, the method of using laser ranging sensor to capture flight height information is proposed to solve the problem that scale information cannot be initialized under small maneuvering conditions in the cruise segment. Through the fusion of inertial/laser ranging/image information, Realize the scale unification of real-time image and reference image; secondly, the method based on projection and quantized gradient direction histogram is used to extract and binary code structural features to solve the problem of mismatching due to inconsistent image information in the process of heterogeneous image matching. A geometric positioning method based on inertial/laser rangefinder, to solve the problem of large attitude error in the process of PNP pose calculation of two-dimensional plane control points during high-altitude flight, this method can achieve stability between heterogeneous images Matching to improve the autonomy of the combined navigation algorithm.

In the present invention, in order to realize the inertial vision combined navigation and positioning based on heterologous image matching, it is first necessary to use the inertial navigation attitude information and the laser ranging height information to perform orthophoto correction on the real-time image and the reference image respectively, and unify the real-time image and the reference image. The scaled size of the graph.

系)、惯导载体坐标系(b系)和地理坐标系(n系)，其各自定义具体如下。Specifically, the coordinate system specifically includes a camera coordinate system (c system), an ortho camera coordinate system (

system), inertial navigation carrier coordinate system (b system) and geographic coordinate system (n system), and their definitions are as follows.

Camera coordinate system (c system): take the principal point of the image side of the optical system as the origin o _c ; when observing the optical system, the x _c axis is parallel to the horizontal axis (long side) of the imaging plane coordinate system, and the left direction is positive ; y _c axis is parallel to the longitudinal axis (short side) of the imaging plane coordinate system, and the downward direction is positive; z _c axis points to the observer, and forms a right-handed coordinate system with x _c axis and y _c axis.

系)：假设空中存在一个正射摄像机，该摄像机生成的图像不需经过校正即为正射影像，可知

坐标系的三轴分别指向东、南、地。Ortho camera coordinate system (

System): Assuming that there is an orthophoto camera in the air, the image generated by the camera is an orthophoto image without correction, and it can be seen that

The three axes of the coordinate system point to east, south, and ground, respectively.

Inertial navigation carrier coordinate system (b system): Inertial navigation is installed on the aircraft carrier, the origin of the coordinate system O _b is taken as the inertial navigation center of mass, X _b is along the longitudinal axis of the carrier, and forward is positive; Y _b is along the vertical axis of the carrier, Up is positive; Z _b is along the side axis of the carrier, and right is positive.

Geographical coordinate system (n system): The origin of the coordinate system O _n is taken as the center of mass of the aircraft, the X _n axis refers to the north, the Y _n axis refers to the sky, and the Z _n axis refers to the east.

投影到图像坐标系中的齐次坐标

的过程可以描述为

According to the pinhole camera model, the spatial point P is from homogeneous coordinates in the world coordinate system

Homogeneous coordinates projected into the image coordinate system

The process can be described as

in,

是一个旋转矩阵，描述了从世界坐标系向摄像机坐标系的旋转过程。

是从世界坐标系向摄像机坐标系转换的平移向量，即世界坐标系原点在摄像机坐标系下的坐标，

为空间点P在世界坐标系中的齐次坐标。In the formula, c and r are the column coordinate value and row coordinate value of the space point P in the image coordinate system, respectively, z _c is the z component of the coordinate of the space point P in the camera coordinate system, f is the principal distance of the optical system, s _c is the distance between adjacent pixels in the column direction of the image sensor, s _r is the distance between adjacent pixels in the upward direction of the image sensor, [c _c ,c _r ] ^T is the main point of the image, and the optical system is based on the needle Designed for the hole camera model, the image sensor is part of the optical system. There are cmos photosensitive components on the image sensor. Each pixel of the image corresponds to a dot matrix of the sensor. The distance between two points is called the pixel size, which is the distance between adjacent pixels.

is a rotation matrix that describes the rotation process from the world coordinate system to the camera coordinate system.

is the translation vector converted from the world coordinate system to the camera coordinate system, that is, the coordinates of the origin of the world coordinate system in the camera coordinate system,

is the homogeneous coordinate of the space point P in the world coordinate system.

这两个摄像机在同一地点不同角度对地面成像，其中

生成的影像为正射影像。根据针孔摄像机成像模型，世界坐标系中的空间点P在两个摄像机中所成的图像坐标分别为Suppose there are two cameras with identical internal parameters, denoted as c and

The two cameras image the ground from different angles at the same location, where

The resulting image is an orthophoto. According to the pinhole camera imaging model, the image coordinates of the space point P in the world coordinate system in the two cameras are respectively

The position and attitude matrix can be transformed as follows

为两个摄像机之间的旋转矩阵，代入

成像方程可得in

For the rotation matrix between the two cameras, substitute

The imaging equation is available

的条件下，可将非正射影像

转化为正射影像

That is, after obtaining the camera's internal parameters K, rotation matrix

Under the condition of , the non-orthophoto image can be

Convert to orthophoto

实现正射影像校正，其中，

为非正射影像，

为正射影像，

为空间点P在世界坐标系中的齐次坐标，z_c为空间点P在摄像机坐标系中的坐标的z分量，f为光学系统主距，s_c为图像传感器上列方向上相邻像素之间的距离，s_r为图像传感器上行方向上相邻像素之间的距离，[c_c,c_r]^T为图像的主点，

为两个内参完全一致的摄像机之间的旋转矩阵，

为从世界坐标系向摄像机坐标系转换的旋转矩阵，

为从世界坐标系向摄像机坐标系转换的平移向量。Therefore, both the real-time graph and the baseline graph can be based on

Implements orthophoto correction, where,

is a non-orthophoto image,

is an orthophoto,

is the homogeneous coordinate of the space point P in the world coordinate system, z _c is the z component of the coordinate of the space point P in the camera coordinate system, f is the principal distance of the optical system, and s _c is the adjacent pixels in the column direction of the image sensor The distance between them, s _r is the distance between adjacent pixels in the upward direction of the image sensor, [c _c ,c _r ] ^T is the main point of the image,

is the rotation matrix between two cameras with identical internal parameters,

is the rotation matrix converted from the world coordinate system to the camera coordinate system,

is the translation vector transformed from the world coordinate system to the camera coordinate system.

来获取，其中，μ为像元尺寸，f为相机焦距，l为相机光心距拍摄点距离。After orthophoto correction is performed on the real-time image and the reference image respectively, the scaling size of the real-time image and the baseline image needs to be unified. In order to unify the pixel resolution of the real-time image and the reference image, both are scaled to 1m/pixel. The benchmark map is directly obtained by commercial map software, and the real-time map is obtained by real-time processing of the distance between the optical center of the camera and the shooting point measured by a laser rangefinder. The zoom factor of the real-time graph and the reference graph can be adjusted according to

to obtain, where μ is the pixel size, f is the focal length of the camera, and l is the distance between the optical center of the camera and the shooting point.

Further, after unifying the scaling sizes of the real-time graph and the reference graph, the first projection and quantized gradient direction histogram of the real-time graph and the second projection and quantized gradient direction histogram of the reference graph can be obtained respectively. In the present invention, the grayscale difference between the heterologous images is relatively large, but the structural features in the images are still relatively close, as shown in Fig. 5(a) and Fig. 5(b). The gradient of the image reflects the structural features in the image well, as shown in Figure 5(c) and Figure 5(d). Do a similarity comparison.

Specifically, in the present invention, acquiring the first projection and quantized gradient direction histogram of the real-time image specifically includes: performing Gaussian filtering on the real-time image; using the Sobel operator to extract the grayscale image gradient feature of the Gaussian filtered real-time image to obtain The gradient image of the real-time graph; based on the gradient image of the real-time graph, the gradient histogram of the real-time graph is counted; the gradient histogram of the real-time graph is projected and quantized to obtain the first projected and quantized gradient direction histogram; the second of the reference graph is obtained The projected and quantified gradient direction histogram specifically includes: performing Gaussian filtering on the reference image; using the Sobel operator to extract the grayscale image gradient features of the Gaussian filtered baseline image to obtain the gradient image of the baseline image; based on the gradient image of the baseline image, statistical The gradient histogram of the benchmark map; the gradient histogram of the benchmark map is projected and quantized to obtain a second projected and quantized gradient direction histogram. The specific process of obtaining the histogram of projected and quantized gradient directions will be described in detail below with respect to the real-time graph or the reference graph.

First, the image is denoised using Gaussian filtering. In image processing, a two-dimensional Gaussian function is used to construct a two-dimensional convolution kernel to process pixel values. The two-dimensional Gaussian function can be described as follows:

其中，P(μ₁,μ₂)为像素点，σ为高斯系数。

Among them, P(μ ₁ , μ ₂ ) is the pixel point, and σ is the Gaussian coefficient.

Represent image local pixels as:

When filtering the image, the convolution operation is performed on each pixel point. For the pixel point P (μ ₁ , μ ₂ ), the convolution kernel calculated by the Gaussian function is used to convolve the pixel point, and the convolution kernel can represent for:

K(μ₁-1,μ₂-1) K(μ₁,μ₂-1) K(μ₁+1,μ₂-1) K(μ₁-1,μ₂) K(μ₁,μ₂) K(μ₁+1,μ₂) K(μ₁-1,μ₂+1) K(μ₁,μ₂+1) K(μ₁+1,μ₂+1)

Among them, K is the result after normalization.

After convolution, the value of the corresponding pixel is:

After this operation is performed on all pixel values of the image, the Gaussian filtered image is obtained. Since the deconvolution is generally performed sequentially, from left to right, and from top to bottom, this process is called convolution. Sliding of the nuclei. When sliding to the boundary, the corresponding position of the convolution kernel has no pixel value, so before performing the convolution, the edge of the original image is usually expanded, and then the convolution operation is performed. The expanded part can be discarded or automatically filled with zeros during convolution.

The Gaussian function has five important properties that make it particularly useful in image processing. These properties show that the Gaussian smoothing filter is a very effective low-pass filter in both the spatial and frequency domains, and in real images used effectively in processing. details as follows:

(1) The two-dimensional Gaussian function has rotational symmetry, that is, the smoothness of the filter is the same in all directions. Generally speaking, the edge direction of an image is not known in advance, so it cannot be determined before filtering. More smoothing is required in one direction than the other. Rotational symmetry means that the Gaussian smoothing filter is not biased in either direction in subsequent edge detections.

(2) The Gaussian function is a single-valued function. This shows that the Gaussian filter replaces the pixel value of the point with the weighted mean of the pixel neighborhood, and the weight of each neighborhood pixel point increases or decreases monotonically with the distance between the point and the center point. This property is very important , because the edge is a local feature of the image, if the smoothing operation still has a great effect on the pixels far away from the center of the operator, the smoothing operation will distort the image.

(3) The Fourier transform spectrum of the Gaussian function is a single lobe. This property is a direct corollary of the fact that the Fourier transform of the Gaussian function is equal to the Gaussian function itself. Images are often contaminated with unwanted high frequency signals (noise and fine texture). The desired image features, such as edges, contain both low-frequency and high-frequency components. The single lobe of the Fourier transform of the Gaussian function means that the smooth image is not polluted by unwanted high-frequency signals, while preserving most of the desired signal.

(4) The width of the Gaussian filter (determining the degree of smoothness) is characterized by parameters, and the relationship with the degree of smoothness is very simple. The larger it is, the wider the frequency band of the Gaussian filter, and the better the smoothness. By adjusting the smoothness parameter, a compromise can be achieved between excessive blurring of image features (oversmoothing) and excessive undesired abrupt changes (undersmoothing) in the smoothed image due to noise and fine textures.

(5) Due to the separability of the Gaussian function, larger size Gaussian filters can be effectively implemented. The two-dimensional Gaussian function convolution can be performed in two steps, first convolving the image with a one-dimensional Gaussian function, and then convolving the convolution result with the same one-dimensional Gaussian function perpendicular to the direction. Therefore, the computational cost of 2D Gaussian filtering grows linearly rather than quadratically with the width of the filter template.

Secondly, after Gaussian filtering of the image, the Sobel operator is used to extract the gradient features of the grayscale image. The specific calculation is as follows:

Gx=(-1)×f(x-1,y-1)+0×f(x,y-1)+1×f(x+1,y-1)

+(-2)×f(x-1,y)+0×f(x,y)+2×f(x+1,y)

+(-1)×f(x-1,y+1)+0×f(x,y+1)+1×f(x+1,y+1)

=[f(x+1,y-1)+2×f(x+1,y)+f(x+1,y+1)]-[f(x-1,y-1)+2× f(x-1,y)+f(x-1,y+1)]

Gy=1×f(x-1,y-1)+2×f(x,y-1)+1×f(x+1,y-1)

+0×f(x-1,y)+0×f(x,y)+0×f(x+1,y)

+(-1)×f(x-1,y+1)+(-2)×f(x,y+1)+(-1)×f(x+1,y+1)

=[f(x-1,y-1)+2×f(x,y-1)+f(x+1,y-1)]-[f(x-1,y+1)+2× f(x,y+1)+f(x+1,y+1)]

梯度方向

梯度方向实际范围为[-ππ]，实际解算的θ范围映射到了

由于图像从亮到暗和从暗到亮引起的图像梯度方向可能为互成补角的关系，这样映射可以保证红外图像不受明暗变换的影响。得到梯度图像之后，统计梯度直方图，并进行投影和量化。where f(x,y) represents the gray value of the image (x,y) point. Gradient magnitude

Gradient direction

The actual range of the gradient direction is [-ππ], and the actual calculated θ range is mapped to

Since the image gradient directions caused by the image from light to dark and from dark to light may be complementary angles, this mapping can ensure that the infrared image is not affected by the light-dark transformation. After the gradient image is obtained, the gradient histogram is counted, projected and quantified.

到

如图8所示，然后根据细胞中每个像素点的梯度方向的大小落在哪个bin，对应的区间计数加一，这样，对细胞单元内每个像素用梯度方向在直方图中进行加权投影(映射到固定的角度范围)，就可以得到这个细胞单元的方向梯度直方图(HOG)。这样统计下来，得到w×h×8维的特征向量记为T_HOG，其中w和h分别为图像的列宽和行宽，并将T_HOG表示为w×h行，8列的矩阵形式。The image is divided into small connected regions, called cells, where the size of the cell is: 8 × 8 pixels. Statistical histograms of directional gradients for each cell unit. The statistical method is: divide the gradient direction into 8 intervals (bins), from

arrive

As shown in Figure 8, then according to which bin the size of the gradient direction of each pixel in the cell falls, the corresponding interval count is incremented by one, so that each pixel in the cell unit is weighted in the histogram with the gradient direction. (mapped to a fixed angular range), the histogram of directional gradients (HOG) for this cell unit can be obtained. In this way, a w×h×8 dimensional feature vector is obtained and denoted as T _HOG , where w and h are the column width and row width of the image, respectively, and T _HOG is represented as a matrix form of w×h rows and 8 columns.

Finally, after obtaining the image feature description, use the projection matrix to project the w×h×8-dimensional feature description, quantize it into a 24-bit binary code, and the projection matrix is:

进行投影，即Use the projection matrix for each row of T _HOG

to project, i.e.

In this way, the gradient direction histogram statistics matrix T' _HOG after projection is obtained, and all the numbers greater than 0 in the matrix are set to 1, and the numbers less than 0 are set to 0, then each row of the matrix is the quantized 24-bit binary After encoding, the projected and quantized gradient direction histogram statistics are obtained.

来获取，其中，d(a,b)为差别度量，f为单调函数，(u,v)为实时图中的像素坐标集合，I(x+u,y+v)表示实时图在参考图(x,y)处的相似性度量值，T为实时图，I为参考图Further, in the present invention, after the projection and quantization gradient direction histograms of the real-time graph and the reference graph are obtained respectively, the first projection and quantization gradient direction histogram and the second projection and quantization can be based on the similarity measurement principle. The similarity of the gradient direction histogram is detected, and the position of the maximum Hamming distance from the first projection and the quantized gradient direction histogram is retrieved in the second projection and quantized gradient direction histogram, and the position of the maximum Hamming distance is Match the location to the image in the base map for the real-time map. Among them, the similarity S(x, y) between the first projected and quantized gradient direction histogram and the second projected and quantized gradient direction histogram can be determined according to

to obtain, where d(a,b) is the difference measure, f is a monotonic function, (u,v) is the set of pixel coordinates in the real-time image, and I(x+u,y+v) indicates that the real-time image is in the reference image The similarity measure at (x, y), T is the real-time image, I is the reference image

Specifically, set the real-time image template pixel coordinate set R _T ={( _{u i} ,vi )|i∈[1,N]}, where N is the number of pixels in the template, and ( _{u i} ,vi ) is Pixel coordinates. Reference reference image R _I ={(x,y)|x∈[0,N _x -1],y∈[0,N _y -1]}, N _x and N _y are the number of columns and rows of the reference image, respectively number. Among them, the template is T, and the reference image is I. The degree of similarity describing T and I is denoted by S(x,y).

Among them, d(a,b) is the difference measure, f is a monotonic function, (u,v) represents the pixel coordinate set of the real-time image, I(x+u,y+v) represents the real-time image in the benchmark image (x, v) y), by changing the value of (x, y), the similarity matrix S is calculated by pixel-by-pixel sliding of the real-time image in the reference image.

When calculating the similarity measure, instead of directly using the gray value of the image, the Hamming distance is used to calculate the difference between the 24-bit binary features generated by the projection:

Among them, BitCount in the above formula is the number of 1 in the string after XOR of binary strings a and b. When d(a,b)=0, it means that a and b are the same, and when d(a,b)=24, it means that a and b have the greatest difference. Obviously d(a,b) is the difference measure.

Binary strings a and b can also be compared by similarity

When S(a,b)=24, it means that a and b are the same, and when S(a,b)=0, it means that the difference between a and b is the largest. Compare all binary strings of two images and sum them. The larger the sum, the higher the similarity between the two images.

其中，h为飞机飞行高度，x和y为相机正下方与拍摄图像间位置关系，r^d为相机拍摄位置中心点在相机坐标系下的投影，rⁿ为r^d在地理坐标系下投影，

为从相机机体坐标系向地理坐标系转换的矩阵。然后，基于观测量构建卡尔曼滤波器以实现无人机的惯性视觉组合导航定位。Further, after obtaining the image matching position of the real-time map in the reference map, the image matching position can be converted to the position of the UAV, and the conversion relationship between the matching position of the observation image and the position of the UAV is as follows:

Among them, h is the flight height of the aircraft, x and y are the positional relationship between the camera directly below and the captured image, r ^d is the projection of the center point of the camera shooting position in the camera coordinate system, rn is the projection of ^{r d} in the geographic coordinate system,

is the matrix to convert from the camera body coordinate system to the geographic coordinate system. Then, a Kalman filter is constructed based on the observations to realize the inertial vision integrated navigation and positioning of the UAV.

l为相机中心点沿光轴到拍摄位置中心点距离。r^d在地理坐标系下投影为

其中

h为飞机飞行高度，x和y为相机正下方与拍摄图像间位置关系，

为从相机机体坐标系(d系)向地理坐标系(n系)转换的矩阵，

为从惯导载体坐标系(b系)向地理坐标系转换(n系)的矩阵，

为从相机机体坐标系(d系)向惯导载体坐标系(b系)转换的矩阵。因此可以利用激光测距获取l、惯导姿态

与事先标定的安装误差角

将图像匹配位置转换到相机所在位置，即为无人机定位点。Specifically, in the present invention, the positional relationship between the positioning point of the UAV and the image matching point is shown in Figure 10. Taking the camera center point as the origin, the camera body coordinate system d (front upper right) is defined, where dx is related to the inertial navigation system. The carrier coordinate system (b system) is fixed, the d system x axis is in the same direction as the b system x axis, the d system y axis and the b system y axis are in the same direction, and the d system z axis and the b system z axis are in the same direction. The rotation angle between the d system and the b system is the installation error between the camera and the inertial navigation system. In the geographic coordinate system n system (North Tiandong), the projection of the center point of the camera shooting position under the camera coordinate system is

l is the distance from the center of the camera to the center of the shooting position along the optical axis. r ^d is projected in the geographic coordinate system as

in

h is the flight height of the aircraft, x and y are the positional relationship between the camera directly below and the captured image,

is the matrix converted from the camera body coordinate system (d system) to the geographic coordinate system (n system),

is the matrix for transforming from the inertial navigation carrier coordinate system (b system) to the geographic coordinate system (n system),

is the matrix converted from the camera body coordinate system (d system) to the inertial navigation carrier coordinate system (b system). Therefore, laser ranging can be used to obtain l, inertial navigation attitude

with the pre-calibrated installation error angle

Convert the image matching position to the position of the camera, which is the positioning point of the drone.

其中，F(t)为t时刻连续状态方程状态转移矩阵，

为t时刻系统随机噪声向量。滤波状态向量包括北天东速度误差(单位：m/s)、纬度误差(单位：rad)、高度误差(单位：m)、经度误差(单位：rad)、北天东向失准角(单位：rad)、相机与惯导间安装误差(单位：rad)和激光测距刻度系数误差，共13维。定义为：Next, a Kalman filter is constructed based on the observations to realize the inertial vision integrated navigation and positioning of the UAV. In the Kalman filter model, the continuous state equation of the system is

Among them, F(t) is the state transition matrix of the continuous state equation at time t,

is the random noise vector of the system at time t. The filtering state vector includes velocity error (unit: m/s), latitude error (unit: rad), altitude error (unit: m), longitude error (unit: rad), and misalignment angle (unit: rad) : rad), installation error between camera and inertial navigation (unit: rad) and laser ranging scale coefficient error, a total of 13 dimensions. defined as:

为北向失准角，

为天向失准角，

为东向失准角，α_x为x轴安装误差，α_y为y轴安装误差，α_z为z轴安装误差，δk为激光测距刻度系数误差。Among them, δV _N is the north speed error, δV _U is the sky speed error, δV _E is the east speed error, δL is the latitude error, δH is the altitude error, δλ is the longitude error,

is the north misalignment angle,

is the misalignment angle of the sky,

is the east misalignment angle, α _x is the x-axis installation error, α _y is the y-axis installation error, α _z is the z-axis installation error, and δk is the laser ranging scale coefficient error.

其中，The system state transition matrix is

in,

为沿x轴相机机体坐标系(b系)相对于惯性坐标系(i系)的加速度在地理坐标系(n系)的投影，

为沿y轴相机机体坐标系(b系)相对于惯性坐标系(i系)的加速度在地理坐标系(n系)的投影，

为沿z轴相机机体坐标系(b系)相对于惯性坐标系(i系)的加速度在地理坐标系(n系)的投影。Among them, V _N is the northing velocity, V _U is the sky velocity, _VE is the easting velocity, R _m , R _n are the two dimensions of the spherical radius, L is the latitude, H is the height, ω _ie is the earth's rotation speed,

is the projection of the acceleration of the camera body coordinate system (b system) relative to the inertial coordinate system (i system) along the x-axis on the geographic coordinate system (n system),

is the projection of the acceleration of the camera body coordinate system (b system) relative to the inertial coordinate system (i system) along the y-axis on the geographic coordinate system (n system),

is the projection of the acceleration of the camera body coordinate system (b system) relative to the inertial coordinate system (i system) along the z-axis on the geographic coordinate system (n system).

The observation equation is defined as follows:

为状态方程，

为观测噪声阵。Among them, H(k) is the observation matrix,

is the equation of state,

for the observation noise array.

Using image matching position and laser ranging as observation information (3D)

Among them, δL is the latitude error, δH is the altitude error, δλ is the longitude error, L _INS is the latitude of the inertial navigation measurement, L _REF is the latitude of the image matching, H _INS is the height of the inertial navigation measurement, and H _REF is the height of the image matching. , λ _INS is the longitude of the inertial navigation measurement, λ _REF is the longitude of the image matching, δL _INS is the latitude error of the inertial navigation measurement, δL _REF is the latitude error of the image matching, δH _INS is the height error of the inertial navigation measurement, and δH _REF is The height error of image matching, δλ _INS is the longitude error of inertial navigation measurement, and δλ _REF is the longitude error of image matching.

in,

δr _l ⁿ is the position error introduced from the image matching position to the position directly below the aircraft according to the laser ranging and inertial navigation attitude information.

为带有误差的图像匹配位置在地理坐标系(n系)下的投影，

为从惯导载体坐标系(b系)向地理坐标系(n系)转换的矩阵，

为带有误差的从惯导载体坐标系(b系)向地理坐标系(n系)转换的矩阵，

为从摄像机坐标系(c系)向惯导载体坐标系(b系)转换的矩阵，

为带有误差的从摄像机坐标系向惯导载体坐标系转换的矩阵，r_l ^c为图像匹配位置在摄像机坐标系(c系)下的投影。where φ is the attitude error angle, η is the installation error angle, r _l ⁿ is the projection of the image matching position in the geographic coordinate system (n system),

is the projection of the image matching position with the error in the geographic coordinate system (n system),

is the matrix converted from the inertial navigation carrier coordinate system (b system) to the geographic coordinate system (n system),

is the matrix converted from the inertial navigation carrier coordinate system (b system) to the geographic coordinate system (n system) with errors,

is the matrix converted from the camera coordinate system (c system) to the inertial navigation carrier coordinate system (b system),

is the matrix converted from the camera coordinate system to the inertial navigation carrier coordinate system with error, and r _l ^c is the projection of the image matching position under the camera coordinate system (c system).

According to formula 2 and formula 3, we can get,

Substitute Equation 4 into Equation 1 to get

Among them, φ _N is the attitude error angle in the north direction, φ _U is the attitude error angle in the sky direction, φ _E is the attitude error angle in the east direction, r _N is the north direction distance, r _U is the sky direction distance, r _E is the east direction distance, η _N is the installation error angle in the north direction, η _U is the installation error angle in the sky direction, and η _E is the installation error angle in the east direction.

From this, the observation matrix H can be written as

H(k)=[H ₁ H ₂ H ₃ H ₄ H ₅ ],

r_N为北向距离，r_U为天向距离，r_E为东向距离。in,

r _N is the distance to the north, r _U is the distance to the sky, and r _E is the distance to the east.

According to another aspect of the present invention, an aircraft is provided, which uses the above-mentioned heterogeneous image matching-based inertial visual integrated navigation and positioning method to perform integrated navigation and positioning. Because the combined navigation and positioning method provided by the present invention can solve the problem of large attitude error in the process of calculating the PNP pose of the two-dimensional plane control point during high-altitude flight, realize stable matching between heterogeneous images, and improve the performance of the combined navigation algorithm. autonomy. Therefore, applying the integrated navigation and positioning method to the aircraft can greatly improve the working performance of the aircraft.

In order to have a further understanding of the present invention, the combined inertial visual navigation and positioning method based on heterologous image matching provided by the present invention will be described in detail below with reference to FIG. 1 to FIG. 10 .

As shown in FIG. 1 to FIG. 10 , according to a specific embodiment of the present invention, an inertial vision combined navigation and positioning method based on heterologous image matching is provided, and the method specifically includes the following steps.

Step 1: Use the inertial navigation attitude information and the laser ranging height information to perform orthophoto correction on the real-time image and the reference image respectively, and unify the scaling size of the real-time image and the baseline image. In this embodiment, there are inconsistencies in perspective and scale between the real-time image and the orthophoto, which cannot be directly used for matching and positioning. Therefore, the inertial navigation attitude information and the laser ranging height information are used for fusion, so as to satisfy the scale-invariant feature While transforming the effect, the matching accuracy is greatly improved.

In step 2, the first projection and quantized gradient direction histogram of the real-time graph and the second projection and quantized gradient direction histogram of the reference graph are obtained respectively. In this embodiment, in computer vision and digital image processing, histogram of gradient orientation (HOG) is a description operator based on shape and edge features that can detect objects. It reflects the edge information of the image target and characterizes the local appearance and shape of the image through the size of the local gradient. However, because the descriptor generation process is lengthy and cannot meet the needs of real-time matching, the gradient features are projected and quantized and encoded into binary descriptions , to improve the matching speed.

Step 3: Detect the similarity between the first projection and the quantized gradient direction histogram and the second projection and the quantized gradient direction histogram based on the similarity measurement principle, and retrieve the similarity between the second projection and the quantized gradient direction histogram and the first projection. and the position of the maximum value of the Hamming distance of the quantized gradient direction histogram, the position of the maximum value of the Hamming distance is the image matching position of the real-time image in the reference image. In this embodiment, after calculating the HOG feature of the real-time image and the reference image respectively, the Hamming distance retrieval method is used to detect the similarity of the two images, and the maximum value of the Hamming distance matching the real-time image is found in the baseline image, It is the position of the real-time graph in the benchmark graph.

Step 4: Convert the image matching position to the position of the UAV and use it as the observation amount. Based on the observation amount, a Kalman filter is constructed to realize the inertial visual integrated navigation and positioning of the UAV. In this embodiment, the position point obtained by image matching is the position captured by the camera. When the aircraft pitches or rolls, the position is not the position of the drone. Therefore, it is necessary to convert the coordinate system to convert the image point from the bottom view. It is calculated that the point directly below the drone is located, and then the visual position information is used as the observation amount to construct a Kalman filter to realize the continuous and autonomous navigation and positioning function.

In summary, the present invention provides an inertial vision combined navigation and positioning method based on heterologous image matching. The method aims at the problem of all-day navigation and positioning of UAVs under the condition of GPS rejection, and develops heterologous image matching and positioning technology. , firstly proposed the method of using laser ranging sensor to capture flight height information to solve the problem that scale information cannot be initialized under small maneuvering conditions in the cruise segment. Through the fusion of inertial/laser ranging/image information, real-time map and benchmark The scale of the image is unified; secondly, the method based on the projection and quantized gradient direction histogram is used to extract and binary code the structural features to solve the problem of mismatching due to inconsistent image information in the process of heterogeneous image matching. The geometric positioning method of laser rangefinder solves the problem of large attitude error in the process of PNP pose calculation of two-dimensional plane control points during high-altitude flight. This method can achieve stable matching between heterogeneous images and improve integrated navigation. Algorithmic autonomy.

For ease of description, spatially relative terms, such as "on", "over", "on the surface", "above", etc., may be used herein to describe what is shown in the figures. The spatial positional relationship of one device or feature shown to other devices or features. It should be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "above" or "over" other devices or features would then be oriented "below" or "over" the other devices or features under other devices or constructions". Thus, the exemplary term "above" can encompass both an orientation of "above" and "below." The device may also be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptions used herein interpreted accordingly.

In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing corresponding components. Unless otherwise stated, the above words have no special meaning and therefore cannot be understood to limit the scope of protection of the present invention.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (9)

1. an inertial vision combined navigation and positioning method based on heterologous image matching, is characterized in that, the described inertial vision combined navigation positioning method based on heterologous image matching comprises: Using inertial navigation attitude information and laser ranging height information to perform orthophoto correction on the real-time image and the reference image respectively, and unify the scaling size of the real-time image and the baseline image; respectively acquiring the first projection and quantized gradient direction histogram of the real-time graph and the second projection and quantized gradient direction histogram of the reference graph; Based on the similarity measurement principle, the similarity between the first projected and quantized gradient direction histogram and the second projected and quantized gradient direction histogram is detected, and the second projected and quantized gradient direction histogram is retrieved from the second projected and quantized gradient direction histogram. the position of the maximum value of the Hamming distance of the first projection and the quantized gradient direction histogram, the position of the maximum value of the Hamming distance is the image matching position of the real-time image in the reference image; The image matching position is converted to the position of the UAV and used as the observation amount, and a Kalman filter is constructed based on the observation amount to realize the inertial visual integrated navigation and positioning of the UAV. 2. The inertial vision combined navigation and positioning method based on heterologous image matching according to claim 1, wherein acquiring the first projection of the real-time image and the quantized gradient direction histogram specifically comprises: performing a step on the real-time image. Gaussian filtering; Using Sobel operator to extract the grayscale image gradient feature of the real-time image after the Gaussian filter to obtain the gradient image of the real-time image; Based on the gradient image of the real-time image, count the gradient histogram of the real-time image; The gradient histogram of the real-time graph is projected and quantized to obtain a first projection and a quantized gradient direction histogram; the obtaining of the second projection and quantized gradient direction histogram of the reference graph specifically includes: performing Gaussian filtering on the reference graph ; Adopt the Sobel operator to extract the grayscale image gradient feature of the benchmark map after the Gaussian filter to obtain the gradient image of the benchmark map; Based on the gradient image of the benchmark map, count the gradient histogram of the benchmark map; The gradient histogram of the reference map is projected and quantized to obtain a second projected and quantized gradient direction histogram. 3. The inertial vision combined navigation and positioning method based on heterologous image matching according to claim 1, wherein the real-time map and the reference map can be based on

Implements orthophoto correction, where,

is a non-orthophoto image,

is an orthophoto,

is the homogeneous coordinate of the space point P in the world coordinate system, z _c is the z component of the coordinate of the space point P in the camera coordinate system, f is the principal distance of the optical system, and s _c is the adjacent pixels in the column direction of the image sensor The distance between them, s _r is the distance between adjacent pixels in the upward direction of the image sensor, [c _c ,c _r ] ^T is the main point of the image,

is the rotation matrix between two cameras with identical internal parameters,

is the rotation matrix converted from the world coordinate system to the camera coordinate system,

is the translation vector transformed from the world coordinate system to the camera coordinate system. 4. The inertial vision combined navigation and positioning method based on heterologous image matching according to claim 3, wherein the scaling coefficient k of the real-time image and the reference image can be determined according to

to obtain, where μ is the pixel size, f is the focal length of the camera, and l is the distance between the optical center of the camera and the shooting point. 5. The inertial vision combined navigation and positioning method based on heterologous image matching according to any one of claims 1 to 4, wherein the first projection and the quantized gradient direction histogram and the second projection and The similarity S(x,y) of the quantized gradient direction histogram can be calculated according to

to obtain, where d(a, b) is the difference measure, f is a monotonic function, (u, v) is the set of pixel coordinates in the real-time image, and I(x+u, y+v) indicates that the real-time image is in the reference image The similarity measure at (x, y), where T is the real-time image and I is the reference image. 6. The inertial vision combined navigation and positioning method based on heterologous image matching according to claim 1, wherein the conversion relationship between the image matching position and the position of the unmanned aerial vehicle is:

Among them, h is the flight height of the aircraft, x and y are the positional relationship between the camera directly under the camera and the captured image, r ^d is the projection of the center point of the camera shooting position in the camera body coordinate system, and r ⁿ is the projection of r ^d in the geographic coordinate system ,

is the matrix to convert from the camera body coordinate system to the geographic coordinate system. 7. The inertial vision integrated navigation and positioning method based on heterologous image matching according to any one of claims 1 to 6, wherein, in the Kalman filter, the state vector comprises a north speed error, a sky speed Error, Easting Speed Error, Latitude Error, Altitude Error, Longitude Error, North Misalignment, Sky Misalignment, Easting Misalignment, x-axis installation error, y-axis installation error, z-axis installation error and laser ranging Scale factor error. 8 . The inertial vision combined navigation and positioning method based on heterologous image matching according to claim 7 , wherein, in the Kalman filter, the observation matrix is H(k)=[H ₁ H ₂ H ₃ . H ₄ H ₅ ], where,

r _N is the distance to the north, r _U is the distance to the sky, and r _E is the distance to the east. 9 . An aircraft, characterized in that, the aircraft uses the heterogeneous image matching-based inertial visual integrated navigation and positioning method according to any one of claims 1 to 8 to perform integrated navigation and positioning.

Images