CN111951331B

CN111951331B - Flight device accurate positioning method and device based on video image and electronic equipment

Info

Publication number: CN111951331B
Application number: CN202010646314.4A
Authority: CN
Inventors: 王勇; 陈东; 干哲; 范梅梅; 李轶博; 陈骁; 肖永辉; 杨伟斌; 王涵; 王晶; 韩晓广; 席有猷; 周青巍
Original assignee: Pla 93114
Current assignee: Pla 93114
Priority date: 2020-07-07
Filing date: 2020-07-07
Publication date: 2024-02-27
Anticipated expiration: 2040-07-07
Also published as: CN111951331A

Abstract

The invention discloses a method and a device for accurately positioning a flying device based on video images and electronic equipment. The method comprises the following steps: acquiring a video image currently shot by a flight device in the flight process; extracting characteristic points in the video image; acquiring image space coordinates of the feature points in an image plane coordinate system; the characteristic points are used as control points, and object space coordinates of the control points in an object space coordinate system are determined according to image space coordinates of the characteristic points in an image plane coordinate system; determining the space position information of a shooting center corresponding to the video image by using a direct linear transformation model; and determining the spatial position information of the shooting center corresponding to the video image as the current position information of the flying device. The method has the advantages that the accurate positioning of the flying device is realized based on the video image shot by the flying device in the flying process, and the cost and the additional weight for positioning the flying device are reduced.

Description

Flight device accurate positioning method and device based on video image and electronic equipment

Technical Field

The invention relates to the technical field of navigation positioning, in particular to a method and a device for accurately positioning a flight device based on video images and electronic equipment.

Background

At present, flying devices such as unmanned aircrafts play a very important role in many fields, and how to accurately acquire the position of the flying device in the flying process is very important for better application of the flying device.

In the related art, position information of a flying device is generally recorded in real time by installing an inertial navigation device or a satellite positioning system on the flying device. However, the installation of the inertial navigation device or the satellite positioning system on the flying device is disadvantageous to the flying of the flying device because the inertial navigation device or the satellite positioning system has a large weight, which increases the weight of the flying device, and also results in a high cost of positioning the flying device because the inertial navigation device or the satellite positioning system has a high cost.

Disclosure of Invention

The present invention aims to solve at least to some extent one of the technical problems in the above-described technology. Therefore, an object of the present invention is to provide a precise positioning method for a flying device based on video images, which solves the technical problems of high cost for positioning the flying device and influence on the flying of the flying device due to the increase of the weight of the flying device in the related art.

The second object of the present invention is to provide a precise positioning device for a flying device based on video images.

A third object of the present invention is to propose an electronic device.

A fourth object of the present invention is to propose a computer readable storage medium.

To achieve the above objective, an embodiment of a first aspect of the present invention provides a method for precisely positioning a flying device based on video images, including the following steps: acquiring a video image currently shot by a flight device in the flight process; extracting feature points in the video image; acquiring image space coordinates of the feature points in an image plane coordinate system; taking the characteristic points as control points, and determining object space coordinates of the control points in an object space coordinate system according to the image space coordinates of the characteristic points in an image plane coordinate system; determining the shooting center space position information corresponding to the video image by using a direct linear transformation model according to the image space coordinates of the feature points in an image plane coordinate system and the object space coordinates of the control points in an object space coordinate system; and determining the space position information of the shooting center corresponding to the video image as the current position information of the flying device.

To achieve the above object, a second aspect of the present invention provides a precise positioning device for a flying device based on video images, including: the first acquisition module is used for acquiring a video image currently shot by the flight device in the flight process; the extraction module is used for extracting the characteristic points in the video image; the second acquisition module is used for acquiring the image space coordinates of the feature points in an image plane coordinate system; the first determining module is used for taking the characteristic points as control points and determining object space coordinates of the control points in an object space coordinate system according to image space coordinates of the characteristic points in an image plane coordinate system; the second determining module is used for determining the shooting center space position information corresponding to the video image by utilizing a direct linear transformation model according to the image space coordinates of the characteristic points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system; and the third determining module is used for determining the space position information of the shooting center corresponding to the video image as the current position information of the flying device.

To achieve the above object, an embodiment of a third aspect of the present invention provides an electronic device, including a memory, and a processor; the processor executes a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the precise positioning method of the flying device based on the video image according to the embodiment of the first aspect of the invention.

To achieve the above object, according to a fourth aspect of the present invention, there is provided a computer readable storage medium storing a computer program which, when executed by a processor, implements a method for accurately positioning a video image-based flying device according to the first aspect of the present invention.

The technical scheme of the embodiment of the invention has the following beneficial effects:

the video image shot in the flight process based on the flight device is realized, the flight device is accurately positioned, and the camera is low in cost and light in weight because the camera is only required to be added, so that the cost for positioning the flight device is reduced, and the increase of the extra weight of the flight device is reduced.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

FIG. 1 is a flow chart of a method for accurately positioning a video image-based flying device according to one embodiment of the present invention;

FIG. 2 is a histogram equalization schematic according to one embodiment of the invention;

FIG. 3 is a schematic diagram of an image convolution operation principle according to an embodiment of the present invention;

FIG. 4 is an exemplary diagram of a template matching classification method according to an embodiment of the invention;

FIG. 5 is a schematic diagram of collinear conditions according to one embodiment of the invention;

FIG. 6 is a schematic diagram of a direct linear transformation principle according to one embodiment of the present invention;

FIG. 7 is a flow chart of a method for accurately positioning a video image-based flying device according to another embodiment of the present invention;

FIG. 8 is a schematic structural view of a video image-based precise positioning device for a flying device according to one embodiment of the present invention; and

fig. 9 is a schematic structural view of an electronic device according to an embodiment of the present invention.

Detailed Description

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.

It will be appreciated that in the related art, the position information of the flying device is typically recorded in real time by installing an inertial navigation device or a satellite positioning system on the flying device. However, the installation of the inertial navigation device or the satellite positioning system on the flying device is disadvantageous to the flying of the flying device because the inertial navigation device or the satellite positioning system has a large weight, which increases the weight of the flying device, and also results in a high cost of positioning the flying device because the inertial navigation device or the satellite positioning system has a high cost.

According to the method, after the video image which is currently shot by the flying device in the flying process is acquired, the characteristic points in the video image can be extracted, then the image side coordinates of the characteristic points in the image plane coordinate system are acquired, the characteristic points are used as control points, the object side coordinates of the control points in the object side space coordinate system are determined according to the image side coordinates of the characteristic points in the image plane coordinate system, the image side coordinates of the characteristic points in the image plane coordinate system and the object side coordinates of the control points in the object side space coordinate system are utilized to determine the shooting center space position information corresponding to the video image, and then the shooting center space position information corresponding to the video image is determined to be the current position information of the flying device.

First, several coordinate systems to which the present application relates will be briefly described.

The image plane coordinate system is a plane rectangular coordinate system of the position of an image point in the image plane, and the origin of coordinates is usually the center point of the image.

The image space coordinate system is a space rectangular coordinate system of the image point in the space position of the image space, and the origin of coordinates can be set according to the requirement.

The object space coordinate system is a coordinate system of an object in a specified space of a measurer, such as the ground, other reference objects and the like, and the origin of the coordinates can be set according to requirements.

The following describes a video image-based accurate positioning method and device for a flight device, an electronic device and a computer-readable storage medium according to embodiments of the present invention with reference to the accompanying drawings.

First, a precise positioning method of a flight device based on video images provided in the present application will be described with reference to fig. 1. Fig. 1 is a flowchart of a method for accurately positioning a video image-based flying device according to one embodiment of the present invention.

As shown in fig. 1, the method for accurately positioning a flight device based on video images according to the embodiment of the invention may include the following steps:

step 101, acquiring a video image currently shot by a flight device in the flight process.

Specifically, the precise positioning method for the video image-based flying device provided by the application can be executed by the precise positioning device for the video image-based flying device, which is hereinafter referred to as a positioning device, wherein the positioning device can be configured in the electronic equipment so as to precisely position the flying device through lower cost and additional weight increase. The electronic device may be any hardware device capable of performing data processing, such as a mobile phone, a computer, and the like. It will be appreciated that the positioning device may be configured in the controller of the flying device or in the ground command center of the flying device, as the application is not limited in this respect.

Specifically, a camera can be configured in the flying device to shoot video images in real time in the flying process of the flying device. In an exemplary embodiment, the camera may be disposed in front of the flying device, and the present application does not limit the location of the camera in the flying device.

Step 102, preprocessing the video image by using an image enhancement technology and/or an image denoising technology.

It can be understood that in the embodiment of the application, after the video image currently shot by the flying device in the flying process is acquired, the video image can be preprocessed first, so that the radiation quality of the video image is improved, and a foundation is laid for accurately positioning the flying device according to the video image. Of course, the captured video image may be directly used to position the subsequent flying device without preprocessing, which is not limited in this application.

The process of preprocessing a video image using image enhancement techniques will be described first.

In an exemplary embodiment, the image enhancement techniques may include an image gray scale transformation technique, a histogram equalization technique, an image sharpening technique, a white balance processing technique, and the like. The present application describes an image gradation conversion technique, a histogram equalization technique, and an image sharpening technique as examples.

Image gray level conversion technology:

the gray level transformation can increase the dynamic range of the image, expand the contrast, make the image clear and the characteristic obvious, and is one of the important means for enhancing the image. The gray scale of the pixel is corrected mainly by utilizing the point operation of the image, the gray scale value of the corresponding output pixel is determined by the gray scale value of the input pixel, and the gray scale value can be regarded as the conversion operation from pixel to pixel without changing the spatial relationship in the image.

The change in the pixel gray level is performed according to a transfer function g1 (x ', y')=t [ f '(x', y ') ] between the gray value of the input image f' (x ', y') and the gray value of the output image g1 (x ', y'). The conversion function has various forms, and in the embodiment of the application, the conversion can be performed by a linear conversion method, as shown in the following formula (1):

g1(x',y')＝T[f'(x',y')]＝A'*f'(x',y')+B' (1)

Wherein, in the formula (1), the parameter A ' is the slope of the linear function, B ' is the intercept of the linear function on the y axis, f ' (x ', y ') represents the gray scale of the input image, and g1 (x ', y ') represents the gray scale of the output image.

In the embodiment of the present application, the gray value of each pixel in the video image that is currently shot may be substituted into formula (1) to obtain the gray value of each pixel after the video image processing, so as to implement gray conversion on the video image.

The video image shot at present is preprocessed by utilizing an image gray level conversion technology, so that the dynamic range of the video image can be enlarged, the contrast ratio is expanded, the video image is clear and has obvious characteristics, the radiation quality of the video image is improved, and a foundation is laid for accurately positioning a flying device according to the video image.

Histogram equalization techniques:

histogram equalization is the process of converting one image into another with equalized histogram by gray level conversion, i.e. with the same number of pixels at each gray level.

The image histogram may represent the distribution of pixel gray values in the image. In general, in order to make an image clear, contrast is increased, image details are highlighted, and it is necessary to make the distribution of image gradation substantially uniform from dark to bright as shown in fig. 2. The histogram equalization technique is a technique of converting an image with uneven histogram distribution (for example, an image with most of pixel gray scales concentrated in a certain segment shown in the upper part of fig. 2) into a new image with even gray scale distribution by a function, and expanding the dynamic range of the gray scale histogram. Wherein the transformation function for histogram equalization is not uniform, it is the integral of the input image histogram, i.e. the cumulative distribution function.

Let the gray scale transformation s ' =f ' (r ') be a discontinuous and micromanipulable function with limited slope, it converts the input image Ii ' (x ', y ') into the output image Io ' (x ', y '), the histogram of the input image is Hi ' (r '), and the histogram of the output image is Ho ' (s '), then the corresponding small area elements after gray scale transformation are equal according to the meaning of the histogram, i.e. there is a relationship as shown in formula (2) between Ho ' (s ') and Hi ' (r ').

Ho'(s')ds'＝Hi'(r')dr' (2)

The mapping relation S 'in the final histogram equalization process can be obtained according to the analysis' _k In the form as shown in equation (3).

Where n 'is the sum of the pixels in the image, n' _j The number of pixels that are the current gray level, L, is the total number of gray levels possible in the image.

In the embodiment of the application, the video image can be subjected to histogram equalization by using the formula (3), so that the processed video image is obtained. The gray distribution of the processed video image is approximately uniform from dark to bright, the processed video image is clearer, the gray contrast of the image is increased, details are enhanced, the radiation quality of the video image shot at present is improved, and a foundation is laid for accurately positioning the flying device according to the video image.

Image sharpening techniques:

The purpose of image sharpening is to sharpen edges, contours, and details of the image, and the root cause of blurring of the smoothed image is that the image is sharpened by performing an inverse operation (e.g., a differential operation) because the image is subjected to an averaging or integration operation. Therefore, the application makes the processed video image clearer by respectively performing differential operation on the video image shot currently.

In an exemplary embodiment, high pass filtering and spatial differentiation may be employed for image sharpening.

It can be understood that, for the image sharpening by the high-pass filtering method, since the edge or the detail (edge) part of the line of the image corresponds to the high-frequency component of the image spectrum, the high-frequency component is smoothly passed through by adopting the high-pass filtering, and the middle-low frequency component is properly restrained, so that the detail of the image can be made clear, and the image sharpening is realized.

In an exemplary embodiment, image sharpening may be implemented based on the laplacian operator. Specifically, the differential operator used may be a laplace operator, which is a two-dimensional second order differential operator and is non-directional, as shown in formula (4).

For example, a 3×3 Laplains convolution template may be:

In this embodiment of the present application, the sharpened video image may be obtained by performing laplace operation on the currently photographed video image f ' (x ', y ') according to the following formula (6).

Where h ' (x ', y ') is the sharpened video image.

The image edge of the video image after sharpening is clearer, so that the radiation quality of the video image is improved, and a foundation is laid for accurately positioning the flying device according to the video image.

The following describes a process of preprocessing a video image using an image denoising technique.

In an exemplary embodiment, the denoising of the video image may be performed by a median filtering technique, a gaussian filtering technique, a bilateral filtering technique, or the like.

Median filtering technique:

the median filtering technique is a nonlinear smoothing technique that sets the gray value of each pixel to the median of the gray values of all pixels within a certain neighborhood window of the point. The median filtering is a nonlinear signal processing technology capable of effectively suppressing noise based on a sequencing statistical theory, and the basic principle of the median filtering is to replace the value of one point in an image with the median value of the values of each point in a neighborhood of the point, so that surrounding pixel values are close to the true value, and the isolated noise point is eliminated.

In particular, each pixel in the video image can be scanned by a two-dimensional sliding template with a certain structure, the pixels covered by the template in the video image are ordered according to the size of the pixel values, a monotonically ascending or descending two-dimensional data sequence is generated, and therefore the median value in the two-dimensional data sequence is used as the value of the pixel point corresponding to the central pixel point of the template in the video image.

Wherein the two-dimensional median filtering can be expressed as shown in formula (7):

g2(x',y')＝med{f'(x'-k',y'-l'),(k',l'∈W)} (7)

where f ' (x ', y ') is the original video image, and g2 (x ', y ') is the processed video image. W is the two-dimensional sliding template, and k 'and l' are the row number and the column number of the pixels in the two-dimensional sliding template, respectively. The two-dimensional sliding template can be 3*3 or 5*5. In addition, the shape of the two-dimensional sliding template may be linear, circular, cross-shaped, circular ring-shaped, or the like, which is not limited in this application.

The method has the advantages that the current shot video image is preprocessed by the median filtering technology, so that the transition of the pixel gray value after the video image is processed is obviously smoothed, the radiation quality of the video image is improved, and a foundation is laid for accurately positioning the flying device according to the video image.

Gaussian filtering technique:

gaussian filtering is a linear smoothing filtering, is suitable for eliminating Gaussian noise, and is widely applied to a noise reduction process of image processing. The gaussian filtering is a process of performing weighted average on the whole image, and the value of each pixel point is obtained by performing weighted average on the pixel point and other pixel values in the neighborhood.

Specifically, when the image processing is performed by using gaussian filtering, as shown in fig. 3, a template (or referred to as convolution or mask) (B1 in fig. 3) may be used to scan each pixel in the image to be processed (A1 in fig. 3), and the weighted average gray value of the pixels in the neighborhood determined by the template is used to replace the value of the pixel corresponding to the pixel point in the center of the template (the pixel point where the five-pointed star in B1) in the image to be processed.

In particular, the video image to be processed may be first smoothed, and its filter function may be determined as a gaussian function G (x ', y') as shown in formula (8) according to human visual characteristics.

Where G (x ', y') is a circularly symmetric function whose smoothing effect is controllable by sigma.

Then, as shown in fig. 3, the image G (x ', y ') (i.e., B1 in fig. 3) and the video image f ' (x ', y ') (i.e., A1 in fig. 3) to be processed may be convolved in a manner shown in formula (9), so that a processed smoothed video image G3 (x ', y ') may be obtained.

g3(x',y')＝f'(x',y')*G(x',y') (9)

Through the mode, the image filtering based on the Gaussian operator can be realized, the pixel gray value transition of the processed video image is smooth, the pixel continuous part is not interrupted, the radiation quality of the video image is improved, and a foundation is laid for the follow-up accurate positioning of the flight device according to the video image.

Bilateral filtering technology:

the bilateral filtering is a filter capable of protecting edges and removing noise, and the filter is composed of two functions, so that the effect of protecting edges and removing noise can be achieved.

One of the functions of the bilateral filter is to determine the filter coefficients from the geometric spatial distance, and the other function is to determine the filter coefficients from the pixel difference. The bilateral filter has the advantages that edge preservation can be performed, compared with wiener filtering or Gaussian filtering which can be obvious in fuzzy edge and has a poor protective effect on high-frequency details, the bilateral filter reduces noise, one Gaussian variance is added to the Gaussian filtering, and the bilateral filter is based on a Gaussian filtering function of spatial distribution, so that pixels far away from the edge are not affected much by pixels near the edge, and preservation of pixel values near the edge is guaranteed.

Specifically, the edge preserving property of bilateral filtering can be realized by combining a space domain function and a value domain kernel function in the convolution process.

The video image shot at present is preprocessed by bilateral filtering, so that the transition of the gray value of the pixel after the video image is processed is flattened, the edge characteristics are well reserved, the radiation quality of the video image is improved, and a foundation is laid for accurately positioning the flying device according to the video image.

In the embodiment of the present application, when preprocessing a video image, only image enhancement processing may be performed on the video image, or only image denoising processing may be performed on the video image, or image enhancement processing and image denoising processing may also be performed on the video image at the same time, and in addition, any image enhancement processing technique may be selected as required to implement image enhancement, or any image denoising processing technique may be selected as required to implement image denoising.

And step 103, extracting characteristic points in the video image.

Wherein the video image in this and subsequent steps is a preprocessed video image.

It can be understood that each frame of video image taken during the flight of the flying device corresponds to a point in time. In the embodiment of the present application, the feature point in the current video image may be a feature point having the same feature as the current video image and a video image corresponding to a time point adjacent to the current video image. For example, assuming that a time point corresponding to a currently shot video image and a current video image (a video image in which feature point extraction is required after preprocessing the currently shot video image) is t00 and an adjacent time point before t00 is t01, the feature point in the current video image may be a feature point having the same feature as the video image corresponding to t01 in the current video image.

It can be understood that, because the scenes shot by the cameras arranged on the flying device are changed at any time during the flying process of the flying device, and the larger the time interval between the time points is, the larger the scene change degree is, therefore, the video images corresponding to adjacent time points may have more feature points with the same features, the video images with far time points may have fewer feature points with the same features, that is, the larger the time interval between the time points is, and the fewer the number of feature points with the same features of the video images corresponding to the time points are.

In an exemplary embodiment, feature points in a video image may be extracted by a template matching classification method, a geometric classifier, an artificial neural network classifier, a support vector machine classifier, and the like.

The method for classifying the samples of the most similar templates is a template matching classification method.

The template matching classification compares an unknown image, i.e., an image to be identified, with a standard image to see if they are identical or to calculate their degree of similarity. The template matching classifier takes each sample of the training sample set as a standard template, compares the image to be identified with each template, finds out the standard template which is the most similar and closest, and takes the nearest category in the standard template as the category of the identification result. In the classifying process, any image to be identified is compared with the existing templates in similarity, or the characteristic of each image to be identified is compared with the average value of the characteristic values of various templates to find out the most similar template.

As shown in fig. 4, the template is set to be T1 (M1, n 1) and the size thereof is m1×m1; the image to be compared is S1 (M1, N1), the size of which is N1 XN 1, and N1 is more than or equal to M1. The template T1 is overlapped on the image S1 to be compared and translated, and the area covered by the template is called sub-image S1 ^i',j' The coordinates of the pixel point in the upper left corner of the template, i ', j', in the image S1, called the reference point, can be seen: and 1 is less than or equal to i ', j' is less than or equal to N-M+1.

Now T1 and S1 can be compared ^i',j' If the two are identical, the difference is zero. In an exemplary embodiment, the degree of similarity (similarity) D (i ', j') thereof may be described using the following formula (10).

Thus, the correlation coefficient R (i ', j') of the following formula (11) can be used as the similarity measure:

the characteristic of each image to be compared can be compared with the average value of the characteristic values of various templates by using the formula (10) or (11) so as to find out the most similar template and realize matching.

In this embodiment of the present application, the current video image and the video image corresponding to the adjacent time point (for example, the adjacent time point before the time point corresponding to the current captured video image) may be compared in a similar manner, and then, according to the similarity and the magnitude of the preset similarity threshold, a point with the similarity greater than the preset threshold is extracted as the feature point of the current video image.

The size of the similarity threshold can be set according to requirements.

It will be appreciated that the smaller the similarity threshold setting, the more feature points of the extracted video image, and the larger the similarity threshold setting, the fewer feature points of the extracted video image, and therefore, the required number of feature points can be obtained by setting the size of the similarity threshold.

And 104, acquiring the image side coordinates of the feature points in the image plane coordinate system.

And 105, taking the characteristic points as control points, and determining the object space coordinates of the control points in the object space coordinate system according to the image space coordinates of the characteristic points in the image plane coordinate system.

And 106, determining the photographic center space position information corresponding to the preprocessed video image by using a direct linear transformation model according to the image space coordinates of the feature points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system.

Specifically, the spatial position information of the photographing center corresponding to the video image after the current preprocessing can be determined by using a direct linear transformation model. Then, before determining the spatial position information of the photographing center corresponding to the video image, a direct linear transformation model needs to be established. The process of establishing the direct linear transformation model may be performed before step 103, or may be performed after step 103 and before step 106, and may be performed before step 106.

It can be understood that the area array video image has the characteristic of central projection, and in the embodiment of the application, the direct linear transformation model can be established based on the characteristic of central projection of the area array video image. For knowledge of the center projection, reference may be made to descriptions in the related art, which are not repeated herein.

Wherein, the spatial position information of the photographing center is used for representing the spatial position of the photographing light beam at the photographing moment, and can comprise three-dimensional coordinate values (X _s ，Y _s ，Z _s )。

It should be noted that, for a frame of video image, the spatial position information of the shooting center of the video image is the spatial position information of a camera configured in the flying device when shooting the video image, that is, the spatial position information of the flying device in a space rectangular coordinate system at the moment corresponding to the frame of video image recorded by the ephemeris of the flying device.

It is understood that the direct linear transformation model in this application is built based on collinear conditions. The principle of a series of problems such as single image space back intersection, double image space front intersection, optical handwriting area network adjustment and the like is based on the collinear condition, and the expression form and the use method of the collinear condition are different according to the specific situation of the processed problem.

The principle of the collinearity condition and the process of obtaining the collinearity condition equation will be described first. In this process, (x, y) is the coordinate of the image point, (x) ₀ ，y ₀ ) Coordinates of the principal point (x) which is the central point of the image ₀ ，y ₀ F) is an intra-azimuth element of the image, (X) _S ,Y _S ,Z _S ) The space coordinates of the object space corresponding to the image point are (X, Y, Z) the space coordinates of the object space corresponding to the image point, and (X) _A ，Y _A ，Z _A ) An object space coordinate which is an object point, (a) _i ，b _i ，c _i ) (i=1, 2, 3) is the 9 directional cosine of the 3 external azimuth elements of the image, (Δx, Δy) is the systematic error correction, which contains ds, dβ.

As shown in fig. 5, S is the imaging center, and the coordinates thereof in a predetermined object space coordinate system are assumed to be (X _s ,Y _s ,Z _s ) A is any object space point, and its object space coordinates are (X _A ,Y _A ,Z _A ). a is the conformation of A on the image, and the corresponding image space coordinates and image space auxiliary coordinates are (X, Y, -f) and (X, Y, Z), respectively. When the three points S, A, a are located on a straight line, the auxiliary coordinates (X, Y, Z) of the image point a and the space coordinates (X) of the object point A _A ,Y _A ,Z _A ) The following relationship is directly present:

as can be seen from the above equation (12), the image space coordinates and the image space auxiliary coordinates have a relationship as shown in the equation (13):

The above formula (13) is developed as:

the above formula (14) is then taken into formula (12) taking into account the coordinates (x) of the principal point ₀ ，y ₀ ) The following formulas (15) and (16) can be obtained.

The above formulas (15) and (16) are collinear conditional equations.

It will be appreciated that the direct linear transformation solution is an algorithm that establishes a direct linear relationship between the coordinates of the image point coordinators and the spatial coordinates of the object side of the corresponding object point. The coordinate of the coordinate instrument refers to a direct reading of the coordinate instrument, namely, the coordinate reading of the coordinate instrument with the main point of the image as the origin is not required to be converted.

The direct linear transformation solution is particularly suitable for photogrammetry processing of images shot by a non-measuring camera because initial approximations of the internal azimuth element and the external azimuth element are not needed. Close-range photogrammetry often uses various types of non-metrology cameras, such as normal cameras, high-speed cameras, etc., and thus the algorithm becomes an important component of close-range photogrammetry.

The direct linear transformation solution is in principle derived from collinear conditional equations.

According to the collineation conditional equations (15) and (16), as shown in fig. 6, when one frame of image taken by a non-measuring camera is placed on a certain spatial coordinate system, the above equations (15) and (16) evolve into the following equations (17) and (18).

The systematic error correction (Δx, Δy) in equations (17) and (18) is assumed to temporarily contain only the linear error correction portion caused by the coordinate system non-perpendicularity error dβ and the scale non-uniformity error ds. The coordinate system c-xy is a non-rectangular coordinate system, and the non-perpendicularity between two coordinate axes is dβ. Two coordinate systems are respectively rectangular coordinate systems with the principal point o as the originAnd non-rectangular coordinates o-xy. The coordinates of the principal point o of the image are (x ₀ ,y ₀ ). The coordinates of the certain image point p' in the non-rectangular coordinate system o-xy are (om ₂ ,om ₁ '), which is affected by dβ and ds to contain a linear error. The point p corresponding to the point p' is the ideal position, which is +.>Coordinates of->No errors are contained. Here->

Assuming that the x-direction has no scale error (the direction scale normalization factor is 1), and the y-direction ratioThe scale normalization coefficient is 1+ds. At this time, if the main distance of the x-direction photo is f _x Then the main distance f of the photo in the y direction is

The scale non-uniform error ds can be considered to be caused by factors such as non-uniform unit lengths of the x axis and the y axis of the used coordinate system, uneven deformation of photographic materials, and the like; while the non-orthogonality error dβ may be considered to be caused by non-perpendicularity of the x-axis and y-axis of the coordinate apparatus used.

Thus, the linear error corrections Δx and Δy are:

Δx＝(1+ds)(y-y ₀ )sindβ≈(y-y ₀ )sindβ (20)

Δy＝[(1+ds)cosdβ-1](y-y ₀ )≈(y-y ₀ )ds (21)

in this case, the collinearly conditional equation including only the linear error correction is in the form shown in equation (22).

l ₄ ＝-(l ₁ X _s +l ₂ Y _s +l ₃ Z _s )

l ₈ ＝-(l ₅ X _s +l ₆ Y _s +l ₇ Z _s )

Wherein r is ₁ ＝-(a ₁ X _S +b ₁ Y _S +c ₁ Z _S )，r ₂ ＝-(a ₂ X _S +b ₂ Y _S +c ₂ Z _S )，r ₃ ＝-(a ₃ X _S +b ₃ Y _S +c ₃ Z _S )。

To sum up, we can derive the basic relation of the direct linear transformation solution:

wherein, the formula (23) is the formula of the direct linear transformation model, l ₁ 、l ₂ ……l ₁₁ Equation coefficients for a direct linear transformation model.

According to l ₁ 、l ₂ ……l ₁₁ Expressions (22) and (23) of the image, the directional cosine (a) of the image can be solved ₃ ,b ₃ ,c ₃ ,a ₂ ) As shown in equation (24).

And then the external orientation element of the image can be obtained:

in summary, for one frame image, we solve for l ₁ 、l ₂ ……l ₁₁ After coefficients, 11 independent parameters of the corresponding image can be solved according to the above relation, wherein the 11 parameters comprise 3 internal azimuth elements (x ₀ ,y ₀ ,f _x ) 6 ectopic elementsAnd a non-orthogonal angle dβ and a scale non-uniform coefficient ds. While the y-direction principal distance f of the image _y Not an independent parameter, since it is f _x And ds, so that independent solution is not needed, and the solution can be obtained by carrying out solution through other parameters.

It is understood that the direct linear transformation solution can also be regarded as a photogrammetric analysis processing method based on a collinear condition equation. The direct linear transformation solution is called because it establishes a direct and linear relationship between the coordinate system coordinates (X, Y) and the object space coordinates (X, Y, Z).

The direct linear transformation can be regarded as a "modified spatial back-to-front-intersection" solution, which "back-intersection" is used to solve for l ₁ 、l ₂ ……l ₁₁ Coefficients whose front intersections are used to solve for the spatial coordinates (X, Y, Z) of the object.

Specifically, after extracting the feature points of the video image after the current preprocessing, the image side coordinates of each feature point in the video image can be determined according to the positions of each feature point in the video image. Then the characteristic points are used as control points, then the object space coordinates of the control points in the object space coordinate system are determined according to the image space coordinates of the characteristic points in the image plane coordinate system, and the object space coordinates of a plurality of characteristic points in the image plane coordinate system and the object space coordinates in the object space coordinate system are substituted into formulas (22) and (23), so that l can be calculated ₁ 、l ₂ ……l ₁₁ Then, according to l ₁ 、l ₂ ……l ₁₁ Values of (2) and formula(24) And (25) can calculate 11 parameters such as the external orientation element and the internal orientation element, and further can calculate the (X _S ,Y _S ,Z _S ) As the photographing center spatial position information.

It should be noted that, in the solution of the intersection behind the traditional space, if the external azimuth element and the internal azimuth element are to be solved at the same time, the control points are strictly prohibited from being arranged in the same plane, otherwise, the solution is unstable. Similarly, in the present application, when the spatial position information of the photographing center is resolved by using the direct linear transformation model, since the external orientation element and the internal orientation element are resolved together, it is also required that the control point cannot be arranged on one plane of any orientation.

In the embodiment of the application, when the direct linear transformation model is utilized to calculate the spatial position information of the photographing center, more than six control points are required to be distributed and controlled, and the control points cannot be arranged on one plane (plane with any azimuth) so as to avoid uncertainty of a calculation result. In an exemplary embodiment, the control points may be uniformly arranged so that they surround the object to be measured, and the larger the conformational range of each control point on the image, the better.

And 107, determining the spatial position information of the shooting center corresponding to the video image as the current position information of the flying device.

The current position information of the flight device can comprise three-dimensional coordinate values of the flight device in a space rectangular coordinate system.

It can be understood that, for a frame of video image, the spatial position information of the shooting center of the video image is the spatial position information of a camera configured in the flying device when shooting the video image, that is, the spatial position information of the flying device in a space rectangular coordinate system at the moment corresponding to the frame of video image recorded by the ephemeris of the flying device. Therefore, after the pre-processed video image corresponding to the space position information of the shooting center is obtained, the space position information of the shooting center corresponding to the video image of the frame can be determined as the current position information of the flying device.

It can be understood that, in the embodiment of the present application, the camera configured in the flying device may also capture video images in real time during the flying process, and send the video images to the positioning device, and then the positioning device performs frame-de-framing processing on the video images to obtain multi-frame video images. That is, in the embodiment of the present application, a plurality of frames of video images may be determined according to video images captured by the flight device during the flight process, and the position information of the flight device at the current time point may be determined according to a frame of video image corresponding to the current time point in the plurality of frames of video images.

Then, step 101 may specifically include:

step 101a, obtaining a video image shot by a flight device in the flight process.

Step 101b, performing a de-framing process on the video image to obtain N frames of video images, where N is a positive integer greater than 1.

Step 101c, determining a frame of video image corresponding to the current time point as the currently shot video image.

It should be noted that, in practical application, the size of N may be set as required.

In addition, in the embodiment of the application, the direct linear transformation model can be applied to any frame of video image to determine the spatial position information of the shooting center corresponding to the any frame of video image, so as to determine the position information of the flying device at the time point corresponding to the any frame of video image.

In one possible implementation form, the photographic center spatial position information corresponding to the N frames of video images respectively may also be determined, and further, the flight trajectory curve of the flight device may be determined according to the photographic center spatial position information corresponding to the N frames of video images respectively, and the landing point position information of the flight device may be determined according to the flight trajectory curve of the flight device.

Specifically, the direct linear transformation model may be utilized to sequentially determine the spatial position information of the photographing center corresponding to each frame of video image in the N frames of video images until the spatial position information of the photographing center corresponding to each frame of video image is determined. And then, performing curve fitting according to the spatial position information of the shooting centers corresponding to the N frames of video images respectively, and determining a flight track curve of the flight device.

It should be noted that, in the embodiment of the present application, N may also be set according to accuracy requirements of landing position information of the flying device, for example, in order to improve accuracy of landing position estimation of the flying device, curve fitting may be performed by using spatial position information of a shooting center corresponding to more video images, so as to improve accuracy of a determined flight trajectory curve of the flying device, and at this time, a value of N may be set to be larger.

In the specific implementation, the photographic center space position information corresponding to the N frames of video images respectively is determined, namely, after the N photographic center space position information is determined, curve fitting can be performed by utilizing the N photographic center space position information so as to determine the flight track curve of the flight device. Because the N frames of video images respectively correspond to one time point, curve fitting can be carried out according to the time points respectively corresponding to the N frames of video images and the space position information of the shooting centers respectively corresponding to the N frames of video images, and the time parameter t when the flying device flies is determined to be an independent variable, and the space position parameter of the flying device is determined to be a flight track curve function of the dependent variable.

In particular, the flight trajectory profile of the flying device may be determined in a number of ways.

Mode one

And performing curve fitting by using a polynomial fitting function according to the corresponding time points of the N frames of video images and the corresponding photographic center space position information, and determining a flight track curve of the flight device.

It can be understood that N frames of video images taken during the flight of the flying device respectively correspond to a time point, and the spatial position information of the photographing center corresponding to the N frames of video images respectively includes three-dimensional coordinate values (X _s ，Y _s ，Z _s ) I.e. coordinate values corresponding to three directions respectively, wherein X _s 、Y _s 、Z _s And respectively representing coordinate values of the flying device in three directions. Then, in the embodiment of the application, the following is advantageousWhen curve fitting is performed by using a polynomial fitting method, the polynomial fitting function may include three polynomials, where each polynomial uses a time parameter t of flight of the flight device as an independent variable, and a coordinate value corresponding to one direction corresponding to the space rectangular coordinate system of the flight device is a dependent variable.

In an exemplary embodiment, according to the time points corresponding to the N frames of video images and the spatial position information of the shooting centers corresponding to the N frames of video images, each coefficient of a polynomial is solved by a general polynomial fitting method, so that a functional formula of a flight trajectory curve of the flight device is determined.

Taking a cubic polynomial as an example, a fitting function of a general polynomial fitting may be in the form shown in formulas (26) - (28).

x1”＝p _x1 +p _x2 t+p _x3 t ² +p _x4 t ³ (26)

y1”＝p _y1 +p _y2 t+p _y3 t ² +p _y4 t ³ (27)

z1”＝p _z1 +p _z2 t+p _z3 t ² +p _z4 t ³ (28)

Wherein p is _x1 、p _x2 、p _x3 、p _x4 、p _y1 、p _y2 、p _y3 、p _y4 、p _z1 、p _z2 、p _z3 、p _z4 And the coefficients are respectively the coefficients of a general polynomial, t is the time parameter of the flight device, and x1 ', y1 ', and z1 ' are respectively coordinate values corresponding to the flight device in three directions of a space rectangular coordinate system.

In an exemplary embodiment, according to the time points corresponding to the N frames of video images and the corresponding spatial position information of the photographing center, each coefficient of the polynomial is solved by using a chebyshev polynomial fitting method, so that a functional formula of a flight trajectory curve of the flight device is determined.

Taking a sixth order polynomial as an example, the fit function of chebyshev polynomial fitting may be in the form shown in formulas (29) - (31).

x2”＝p _x1 +p _x2 t+p _x3 t ² +p _x4 t ³ +p _x5 t ⁴ +p _x6 t ⁵ +p _x7 t ⁶ (29)

y2”＝p _y1 +p _y2 t+p _y3 t ² +p _y4 t ³ +p _y5 t ⁴ +p _y6 t ⁵ +p _y7 t ⁶ (30)

z2”＝p _z1 +p _z2 t+p _z3 t ² +p _z4 t ³ +p _z5 t ⁴ +p _z6 t ⁵ +p _z7 t ⁶ (31)

Wherein p is _x1 、p _x2 、p _x3 、……p _z5 、p _z6 、p _z7 And the coefficients are respectively the coefficients of Chebyshev polynomials, t is the time parameter of flight of the flight device, and x2 ', y2 ' and z2 ' are respectively coordinate values corresponding to the flight device in three directions of a space rectangular coordinate system.

Mode two

And performing curve fitting by using a global optimization method according to the corresponding time points of the N frames of video images and the corresponding spatial position information of the photographing center, and determining a flight track curve of the flight device.

In an exemplary embodiment, automatic best fit function matching can be performed through a Levenberg-Marquardt method and a general global optimization method to obtain a best fit function form, curve fitting is performed through the best fit function, and coefficients of the fit function are solved to determine a flight trajectory curve of the flight device.

A series of fitting function forms can be obtained by carrying out best fitting function matching through a Marquardt method and a general global optimization method, and the embodiment of the application is illustrated by taking a polynomial form as an example. The fitting function may include three polynomials, where each polynomial uses a time parameter t of flight of the flight device as an independent variable and uses a coordinate value corresponding to one direction corresponding to the space rectangular coordinate system of the flight device as a dependent variable. Wherein at least one of At least one term of the polynomials may be an exponential function of a natural constant e, e.g. e ^t 。

In an exemplary embodiment, the form of the fitting function obtained by performing best fit function matching by the marquardt method and the general global optimization method may be the form of formulas (32) - (34).

x3”＝p _x1 +p _x2 t ² +p _x3 t ^0.5 +p _x4 e ^-t (32)

y3”＝p _y1 +p _y2 t+p _y3 t ² +p _y4 t ^0.5 +p _y5 e ^t (33)

z3”＝p _z1 +p _z2 t+p _z3 t ^1.5 +p _z4 t ² +p _z5 t ^2.5 (34)

Wherein p is _x1 、p _x2 、p _x3 、……p _z3 、p _z4 、p _z5 And the coefficients are respectively polynomial coefficients, t is a time parameter of flight of the flight device, and x3 ', y3 ', and z3 ' are coordinate values corresponding to the flight device in three directions of a space rectangular coordinate system.

The curve fitting process is specifically performed according to the spatial position information of the photographing center corresponding to the N frames of video images, and may refer to the description in the related art, which is not described in detail in this application.

Furthermore, the landing point position information of the flying device can be determined according to the flying trace curve.

Specifically, the landing time of the flying device can be acquired first, and then the landing position information of the flying device can be determined according to the landing time and the flying track curve.

More specifically, in the flight process of the flight device, the flight speed and the flight distance of the flight device can be obtained in real time, so that the landing time of the flight device is estimated according to the flight speed and the flight distance of the flight device. After the landing time of the flying device is estimated, the landing time can be substituted into a curve function of the flying track curve to determine landing position information of the flying device.

According to the precise positioning method for the flying device based on the video image, firstly, the video image shot by the flying device in the flying process is acquired, then the video image is preprocessed by utilizing an image enhancement technology and/or an image denoising technology, then the image space coordinates of the feature points in the image plane coordinate system are acquired, then the feature points are used as control points, the object space coordinates of the control points in the object space coordinate system are determined according to the image space coordinates of the feature points in the image plane coordinate system, and then the shooting center space position information corresponding to the preprocessed video image is determined by utilizing a direct linear transformation model according to the image space coordinates of the feature points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system. Therefore, the video image shot in the flight process based on the flight device is realized, the flight device is accurately positioned, and the camera is low in cost and light in weight as the camera is only required to be added, so that the cost for positioning the flight device is reduced, and the increase of the extra weight of the flight device is reduced.

The following describes a method for accurately positioning a flight device based on a video image provided in the present application with reference to fig. 7. Fig. 7 is a flowchart of a method for accurately positioning a video-image-based flying device according to another embodiment of the present invention.

As shown in fig. 7, the method for accurately positioning a flight device based on video images according to the embodiment of the invention may further include the following steps:

step 201, obtaining a video image currently shot by a flight device in the flight process.

Step 202, extracting feature points in a video image.

It can be understood that each frame of video image taken during the flight of the flying device corresponds to a point in time. In the embodiment of the present application, the feature points in the currently captured video image may be feature points having the same feature as the video image corresponding to the time point adjacent to the currently captured video image. For example, assuming that the time point corresponding to the currently photographed video image is t00 and the adjacent time point before t00 is t01, the feature point in the currently photographed video image may be a feature point having the same feature as the video image corresponding to t01 in the currently photographed video image.

In an exemplary embodiment, feature points in the video image may be extracted by a template matching classification method, a geometric classifier, an artificial neural network classifier, a support vector machine classifier, or the like.

In step 203, the image space coordinates of the feature points in the image plane coordinate system are obtained.

In step 204, the feature points are used as control points, and the object space coordinates of the control points in the object space coordinate system are determined according to the image space coordinates of the feature points in the image plane coordinate system.

Step 205, determining the shooting center space position information corresponding to the video image by using a direct linear transformation model according to the image space coordinates of the feature points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system.

Specifically, after extracting the feature points of the video image, the image side coordinates of each feature point in the video image can be determined according to the positions of each feature point in the video image. Then the characteristic points can be used as control points, and then the image space in the image plane coordinate system is used for sitting according to the characteristic pointsThe object coordinates of the control points in the object space coordinate system are determined, and the object coordinates of the plurality of characteristic points in the image plane coordinate system and the object coordinates in the object space coordinate system are substituted into formulas (22) and (23) to obtain the l by calculation ₁ 、l ₂ ……l ₁₁ Then, according to l ₁ 、l ₂ ……l ₁₁ And formulas (24) and (25), can calculate 11 parameters such as the external orientation element and the internal orientation element, and can further calculate the value (X _S ,Y _S ,Z _S ) As the photographing center spatial position information.

In the embodiment of the present application, the spatial position information of the photographing center corresponding to the video image may be determined by any other method, which is not limited in this application.

And 206, determining the spatial position information of the shooting center corresponding to the video image as the current position information of the flying device.

It can be understood that, for a frame of video image, the spatial position information of the shooting center of the video image is the spatial position information of a camera configured in the flying device when shooting the video image, that is, the spatial position information of the flying device in a space rectangular coordinate system at the moment corresponding to the frame of video image recorded by the ephemeris of the flying device. Therefore, the method and the device acquire the photographic center space position information corresponding to the preprocessed video image, and can determine the photographic center space position information corresponding to the video image frame as the current position information of the flying device.

It should be noted that, for details not disclosed in the video image-based accurate positioning method for a flight device in the embodiment of the present invention, please refer to details disclosed in the video image-based accurate positioning method for a flight device in the above embodiment of the present invention, which are not described here again.

According to the precise positioning method for the flying device based on the video image, firstly, the video image shot by the flying device in the flying process is acquired, then, the image side coordinates of the feature points in the image plane coordinate system are acquired, then, the feature points are used as control points, the object side coordinates of the control points in the object side space coordinate system are determined according to the image side coordinates of the feature points in the image plane coordinate system, and then, the shooting center space position information corresponding to the preprocessed video image is determined by utilizing a direct linear transformation model according to the image side coordinates of the feature points in the image plane coordinate system and the object side coordinates of the control points in the object side space coordinate system. Therefore, the video image shot in the flight process based on the flight device is realized, the flight device is accurately positioned, and the camera is low in cost and light in weight as the camera is only required to be added, so that the cost for positioning the flight device is reduced, and the increase of the extra weight of the flight device is reduced.

Fig. 8 is a schematic structural view of a precise positioning device for a video image-based flying device according to an embodiment of the present invention.

As shown in fig. 8, the precise positioning device 100 for a video image-based flying device according to the embodiment of the present invention includes a first acquisition module 11, an extraction module 12, a second acquisition module 13, a first determination module 14, a second determination module 15, and a third determination module 16.

The first acquiring module 11 is configured to acquire a video image currently captured by the flying device in a flight process;

an extracting module 12, configured to extract feature points in the video image;

a second obtaining module 13, configured to obtain an image space coordinate of the feature point in an image plane coordinate system;

a first determining module 14, configured to determine, using the feature point as a control point, an object space coordinate of the control point in an object space coordinate system according to an image space coordinate of the feature point in the image plane coordinate system;

the second determining module 15 is configured to determine, according to the image space coordinates of the feature points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system, the photographing center space position information corresponding to the video image by using a direct linear transformation model;

and the third determining module 16 is configured to determine the spatial position information of the shooting center corresponding to the video image as current position information of the flying device.

Specifically, the precise positioning device for the flight device based on the video image, which is provided by the application, is simply referred to as a positioning device, and the precise positioning method for the flight device based on the video image, which is provided by the application, can be executed. The positioning device can be configured in the electronic equipment to accurately position the flying device through lower cost and additional weight increase. The electronic device may be any hardware device capable of performing data processing, such as a mobile phone, a computer, and the like. It will be appreciated that the positioning device may be configured in the controller of the flying device or in the ground command center of the flying device, as the application is not limited in this respect.

In one embodiment of the present invention, the extraction module 12 is specifically configured to:

and extracting characteristic points in the video image by a template matching classification method.

In an embodiment of the present invention, the positioning device may further include:

and the establishing module is used for establishing a direct linear transformation model according to the characteristics of the central projection of the area array video image.

It should be noted that, for details not disclosed in the precise positioning device for the video image-based flying device in the embodiment of the present invention, please refer to details disclosed in the precise positioning method for the video image-based flying device in the above embodiment of the present invention, and details are not described here again.

According to the precise positioning device of the flying device based on the video image, firstly, the video image shot by the flying device in the flying process is acquired, then, the image side coordinates of the feature points in the image plane coordinate system are acquired, then, the feature points are used as control points, the object side coordinates of the control points in the object side space coordinate system are determined according to the image side coordinates of the feature points in the image plane coordinate system, and then, the shooting center space position information corresponding to the preprocessed video image is determined by utilizing a direct linear transformation model according to the image side coordinates of the feature points in the image plane coordinate system and the object side coordinates of the control points in the object side space coordinate system. Therefore, the video image shot in the flight process based on the flight device is realized, the flight device is accurately positioned, and the camera is low in cost and light in weight as the camera is only required to be added, so that the cost for positioning the flight device is reduced, and the increase of the extra weight of the flight device is reduced.

In order to implement the above embodiment, the present invention further proposes an electronic device 200, as shown in fig. 9, where the electronic device 200 includes a memory 21 and a processor 22. The processor 22 executes a program corresponding to the executable program code by reading the executable program code stored in the memory 21, for implementing the above-mentioned precise positioning method of the flying device based on the video image.

According to the electronic equipment provided by the embodiment of the invention, the processor executes the computer program stored on the memory, so that the accurate positioning of the flying device based on the video image shot by the flying device in the flying process can be realized, and the cost of the camera is low and the weight is light as only the camera is needed to be added, thereby reducing the cost of positioning the flying device and reducing the increase of the extra weight of the flying device.

In order to achieve the above embodiments, the present invention further proposes a computer readable storage medium storing a computer program which, when executed by a processor, implements the above-mentioned video image-based accurate positioning method for a flying device.

The computer readable storage medium of the embodiment of the invention can realize the accurate positioning of the flying device based on the video image shot by the flying device in the flying process by storing the computer program and executing the computer program by the processor, and the cost of the camera is low and the weight is light because the camera is only needed to be added, thereby reducing the cost of positioning the flying device and reducing the increase of the extra weight of the flying device.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the device or element being referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.

In the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "secured," and the like are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communicated with the inside of two elements or the interaction relationship of the two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.

In the present invention, unless expressly stated or limited otherwise, a first feature "up" or "down" a second feature may be the first and second features in direct contact, or the first and second features in indirect contact via an intervening medium. Moreover, a first feature being "above," "over" and "on" a second feature may be a first feature being directly above or obliquely above the second feature, or simply indicating that the first feature is level higher than the second feature. The first feature being "under", "below" and "beneath" the second feature may be the first feature being directly under or obliquely below the second feature, or simply indicating that the first feature is less level than the second feature.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims

1. The precise positioning method of the flying device based on the video image is characterized by comprising the following steps of:

Acquiring a video image, a flight speed and a flight distance shot by a flight device in the flight process, performing frame decomposition processing on the video image to acquire N frames of video images, and determining a frame of video image corresponding to a current time point as a current shot video image;

extracting feature points in the video image;

acquiring image space coordinates of the feature points in an image plane coordinate system;

taking the characteristic points as control points, and determining object space coordinates of the control points in an object space coordinate system according to the image space coordinates of the characteristic points in an image plane coordinate system;

according to the image space coordinates of the characteristic points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system, utilizing a direct linear transformation model to sequentially determine the shooting center space position information corresponding to the N frames of video images;

according to the photographic center space position information corresponding to the N frames of video images and the time points corresponding to the N frames of video images, performing automatic best fit function matching by using a maquardt method and a global optimization method to obtain a best fit function, determining a flight track curve of the flight device according to the best fit function, estimating the landing time of the flight device according to the flight speed and the flight distance, and determining the current position information of the flight device according to the best fit function and the landing time, wherein the best fit function has the following expression:

Wherein,、/>、/>、……、/>、/>、/>coefficients of polynomials respectively, t being the time parameter of flight of the flying device, +.>、/>、/>Coordinate values corresponding to the three directions of the flight device in the space rectangular coordinate system are respectively obtained.

2. The method of claim 1, wherein the extracting feature points in the video image comprises:

and extracting the characteristic points in the video image by a template matching classification method.

3. The method according to any one of claims 1-2, wherein before determining the photography center spatial location information corresponding to the video image using the direct linear transformation model, further comprises:

and establishing the direct linear transformation model according to the characteristics of the central projection of the area array video image.

4. Accurate positioner of flight device based on video image, its characterized in that includes:

the first acquisition module is used for acquiring video images, flight speed and flight distance shot by the flight device in the flight process, performing frame decomposition processing on the video images to acquire N frames of video images, and determining one frame of video image corresponding to the current time point as the current shot video image;

the extraction module is used for extracting the characteristic points in the video image;

The second acquisition module is used for acquiring the image space coordinates of the feature points in an image plane coordinate system;

the first determining module is used for taking the characteristic points as control points and determining object space coordinates of the control points in an object space coordinate system according to image space coordinates of the characteristic points in an image plane coordinate system;

the second determining module is used for sequentially determining the shooting center space position information corresponding to the N frames of video images by utilizing a direct linear transformation model according to the image space coordinates of the characteristic points in the image plane coordinate system and the object space coordinates of the control points in the object space coordinate system;

the third determining module is configured to perform automatic best fit function matching by using a maquardt method and a global optimization method according to the spatial position information of the photographing center corresponding to the N frames of video images and the time point corresponding to the N frames of video images, so as to obtain a best fit function, determine a flight trajectory curve of the flight device according to the best fit function, estimate a landing time of the flight device according to the flight speed and the flight distance, and determine the current position information of the flight device according to the best fit function and the landing time, where an expression of the best fit function is as follows:

5. The apparatus of claim 4, wherein the extraction module is specifically configured to:

6. The apparatus of any one of claims 4-5, further comprising:

and the establishing module is used for establishing the direct linear transformation model according to the characteristics of the central projection of the area array video image.

7. An electronic device, comprising a memory and a processor;

wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory for implementing the video image-based accurate positioning method of a flying device as claimed in any one of claims 1 to 3.

8. A computer readable storage medium storing a computer program, which when executed by a processor, implements the video image-based accurate positioning method of a flying device according to any one of claims 1-3.