WO2019100219A1

WO2019100219A1 - Output image generation method, device and unmanned aerial vehicle

Info

Publication number: WO2019100219A1
Application number: PCT/CN2017/112202
Authority: WO
Inventors: 张明磊; 马岳文
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2017-11-21
Filing date: 2017-11-21
Publication date: 2019-05-31
Also published as: CN109076173A

Abstract

Provided by embodiments of the present invention are an output image generation method, device and unmanned aerial vehicle, the method comprising: obtaining a first image captured by a first capturing device that is carried by an aircraft and has a field of view (FOV) greater than or equal to a preset threshold and a second image captured by a second capturing device carried by the aircraft, and on the basis of a preset algorithm, calculating a position and orientation of the first capturing device when the first image is captured and a position and orientation of the second capturing device when the second image is captured; generating a terrain surface on the basis of the first image and the position and orientation of the first capturing device when the first image is captured; and projecting and splicing the second image on the terrain surface on the basis of the position and orientation of the second capturing device when the second image is captured to obtain an output image. The method, device and unmanned aerial vehicle provided by embodiments of the present invention may improve the accuracy of the output image.

Description

Output image generation method, device and drone

Technical field

The present application relates to the field of UAV application technologies, and in particular, to an output image generation method, device, and drone.

Background technique

Digital Orthophoto Map (DOM) is a digital aerial image/remote sensing image (monochrome/color) that is scanned and processed by digital elevation model. The projection difference is corrected by pixel, and then image mosaic. The image generated by stitching according to the range of the frame. Since the image uses a real terrain surface as a mosaic projection surface, it has real geographic coordinate information, and the true distance can be measured on the image.

In the production of digital orthophotos, a higher flying height is usually used in order to obtain a more orthographic aerial image. In order to ensure a higher ground resolution while using a higher flying height, a longer focal length is required. Camera (small angle of view), but according to the principle that the accuracy of the front intersection is related to the intersection angle (the greater the intersection angle is, the higher the accuracy is in a certain range), the intersection angle of the object point obtained by the telephoto camera intersection is smaller. The geometric accuracy is low, especially in the elevation direction. The inaccuracy in the elevation direction will result in the accuracy of the final digital orthophoto image, and it is necessary to ensure that the image captured by the telephoto camera meets a high overlap rate and needs to shoot a large number of images. Calculated the cost.

Summary of the invention

The embodiment of the invention provides an output image generation method, device and an unmanned person to improve the accuracy and precision of the output image.

A first aspect of the present invention provides a method for generating an output image, including:

Obtaining a first image obtained by the first photographing device carried by the aircraft, and acquiring a second image obtained by the second photographing device mounted on the aircraft, wherein the first photographing device has a field of view angle FOV greater than or equal to a preset threshold ;

Calculating a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when photographing the second image, based on a preset algorithm;

Generating a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured;

And based on the position and posture of the second photographing device when the second image is captured, the second image is projected and stitched on the terrain surface to obtain an output image.

A second aspect of the present invention provides a method for generating an output image, including:

Acquiring a first image obtained by the first photographing device carried by the aircraft, and obtaining a second image obtained by the second photographing device mounted on the aircraft, wherein the FOV of the first photographing device is greater than or equal to a preset threshold, wherein a photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground;

A third aspect of the embodiments of the present invention provides a ground station, including:

a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or Equal to the preset threshold;

The processor is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured, and a position and a posture of the second photographing device when the second image is photographed;

The processor is configured to generate a terrain surface based on a position and a posture of the first image and the first photographing device when the first image is captured;

The processor is further configured to: perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.

A fourth aspect of the embodiments of the present invention provides a ground station, including:

The communication interface is configured to: acquire a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or Is equal to a preset threshold, wherein a photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground;

The processor is configured to: perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.

A fifth aspect of the embodiments of the present invention provides an aircraft controller, including:

A sixth aspect of the embodiments of the present invention provides an aircraft controller, including:

A seventh aspect of the embodiments of the present invention provides a computer readable storage medium, comprising instructions, when executed on a computer, causing a computer to execute the output image generating method according to the first aspect or the second aspect described above

An eighth aspect of the embodiments of the present invention provides a drone, including:

body;

a power system mounted to the fuselage for providing flight power;

a first photographing device and a second photographing device are mounted on the body for capturing an image, wherein an FOV of the first photographing device is greater than or equal to a preset threshold;

And the aircraft controller as described above.

In the embodiment of the present invention, the first image obtained by the first photographing device with the FOV of the aircraft being greater than or equal to the preset threshold is acquired, and the second image obtained by the second photographing device carried by the aircraft is obtained, and based on a preset algorithm, Calculating a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when the second image is photographed; and determining a position and a posture when the first image is taken based on the first image and the first photographing device Generating a terrain surface; and based on the position and posture of the second photographing device when the second image is captured, the second image is projected and stitched on the surface of the terrain to obtain an output image. Due to the large FOV of the first photographing device in the embodiment of the present invention Or equal to a preset threshold, and the larger the FOV, the higher the accuracy of the elevation surface (ie, the terrain surface) obtained based on the image fitting captured by the first photographing device, thereby projecting images taken by other photographing devices on the aircraft to the The corresponding high-precision orthophotos can be obtained on the elevation surface, which improves the accuracy of generating orthophotos.

DRAWINGS

1 is a flowchart of a method for generating an output image according to the present invention;

2 is a schematic diagram of a connection between a ground station and an aircraft according to an embodiment of the present invention;

3a and 3b are schematic diagrams showing output images of two identical scenes provided by the present invention;

4a and 4b are schematic diagrams of output images in two identical scenarios according to an embodiment of the present invention;

FIG. 5 is a flowchart of a method for generating an output image according to an embodiment of the present invention;

FIG. 6 is a flowchart of a method for generating an output image according to an embodiment of the present invention;

Figure 7a is an unspliced near-infrared image obtained by a near-infrared camera;

Figure 7b is a near-infrared output image obtained after the near-infrared image shown in Figure 7a is obtained by ortho-splicing;

Figure 7c is a visible light output image corresponding to a visible light camera photographed synchronously with the near infrared camera of Figure 7a;

7d is an NVDI index map calculated by using a near-infrared image and a red band image of a visible light output image;

7e is a schematic diagram showing the result of pseudo color rendering using green for the near-infrared image after ortho-splicing;

FIG. 8 is a flowchart of a method for generating an output image according to an embodiment of the present invention;

FIG. 9 is a flowchart of a method for generating an output image according to an embodiment of the present invention;

10a-10b are schematic diagrams of two shooting intervals provided by an embodiment of the present invention;

11 is a schematic structural diagram of a ground station according to an embodiment of the present invention;

12 is a schematic structural diagram of a ground station according to an embodiment of the present invention;

FIG. 13 is a schematic structural diagram of an aircraft controller according to an embodiment of the present invention;

FIG. 14 is a schematic structural diagram of an aircraft controller according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly described with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

It should be noted that when a component is referred to as being "fixed" to another component, it can be directly on the other component or the component can be present. When a component is considered to "connect" another component, it can be directly connected to another component or possibly a central component.

All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. The terminology used in the description of the present invention is for the purpose of describing particular embodiments and is not intended to limit the invention. The term "and/or" used herein includes any and all combinations of one or more of the associated listed items.

Some embodiments of the present invention are described in detail below with reference to the accompanying drawings. The features of the embodiments and examples described below can be combined with each other without conflict.

Embodiments of the present invention provide an output image generation method, which may be performed by a ground station or a controller mounted on a drone. The following embodiment is a detailed description of the ground station. The implementation manner of the controller is similar to that of the ground station, and is not described in this embodiment. Referring to FIG. 1 , FIG. 1 is a flowchart of a method for generating an output image according to the present invention. As shown in FIG. 1 , the method in this embodiment includes:

Step 101: Obtain a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where a field of view (FOV) of the first photographing device is greater than Or equal to the preset threshold.

The ground station in this embodiment is a device having a computing function and/or processing capability, and the device may specifically be a remote controller, a smart phone, a tablet computer, a laptop computer, a watch, a wristband, and the like, and combinations thereof.

The aircraft in this embodiment may specifically be a drone equipped with a photographing device, a helicopter, a manned fixed-wing aircraft, a hot air balloon, or the like.

In this embodiment, the first photographing device may be a photographing device whose FOV is greater than or equal to a preset threshold (for example, a wide-angle camera with an FOV greater than or equal to a preset threshold). The size of the preset threshold may be set according to requirements, which is not limited in this embodiment. Optionally, the visible light image is taken when the first photographing device is a wide-angle camera.

Optionally, the FOV of the second photographing device is smaller than the FOV of the first photographing device. For example, the second photographing device may be a photographing device with a FOV smaller than the preset threshold (for example, a telephoto camera with a FOV smaller than a preset threshold). Alternatively, the second photographing device may also be a near-infrared camera or an infrared camera, and when the second photographing device is a near-infrared camera or an infrared camera, its FOV may be greater than, less than, or equal to the FOV of the first photographing device. Optionally, when the second photographing device is a near-infrared camera, the second photographing device captures the near-infrared image, when the second photographing device is an infrared camera, the infrared image is captured, and when the second photographing device is the telephoto camera, the visible image is captured. .

As shown in FIG. 2, the ground station 21 and the aircraft 22 can be connected through an Application Programming Interface (API) 23, but are not limited to being connected through an API. Specifically, the ground station 21 and the aircraft 22 can be connected by wire or wirelessly, for example, by connecting: WIreless-Fidelity (WI-FI), Bluetooth, software defined radio (software defined radio, Referred to as SDR) or other custom protocols.

Optionally, in this embodiment, the aircraft can perform automatic cruising and photographing according to a predetermined route, and can also perform cruising and photographing under the control of the ground station.

Optionally, in this embodiment, the first photographing device and the second photographing device may perform shooting according to a preset fixed shooting interval (photographing time or shooting distance), or may be based on a relative flying height between the aircraft and the surface according to a preset strategy. Shooting at a suitable shooting interval, such as when the flying height of the aircraft from the surface is high, using a relatively large shooting interval, when the flying height of the aircraft from the surface is low, using a relatively small shooting interval Shooting. Thereby ensuring that the preset image coincidence ratio is satisfied between the images taken at the adjacent time. Of course, this is only an example, and the actual scene is not limited to ensuring the image reclosing ratio by the above method.

Optionally, in the embodiment, the ground station can obtain images obtained by the first photographing device and the second photographing device by using the following possible manners:

In one possible way, the aircraft will make the first shot through its API with the ground station. The images obtained by the camera and the second shooting device are sent to the ground station in real time.

In another possible manner, the aircraft transmits the images obtained by the first photographing device and the second photographing device in a preset time interval to the ground station according to a preset time interval.

In yet another possible manner, after the cruise is over, the aircraft collectively transmits the images obtained by the first photographing device and the second photographing device during the entire cruise to the ground station.

Specifically, based on the above manner, the aircraft may transmit the images captured by the first photographing device and the second photographing device to the ground station in the form of code stream data, or may be sent to the ground station in the form of thumbnails, but according to the aircraft and the ground. The computing power of the station is not specifically limited to the resolution of the returned stream data or thumbnails, and may be the original image. In this embodiment, taking the form of a thumbnail as an example, when the image is sent to the ground station in the form of a thumbnail, the ground station can display the received thumbnail so that the user can clearly see the image obtained by the real-time shooting. .

Step 102: Calculate a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when the second image is captured, according to a preset algorithm.

Optionally, in a first possible implementation, the ground station may calculate a position and a posture when the first imaging device captures the first image, and a second preset image processing algorithm, based on the first preset image processing algorithm. The position and posture of the second photographing device when the second image is taken are calculated. The first preset image processing algorithm and the second preset processing algorithm may be the same or different. Optionally, the first preset image processing algorithm and the second preset processing algorithm in the embodiment may be any of the following algorithms. One: air triangulation, motion recovery structure (SFM) algorithm, real-time location and map construction (SLAM) algorithm, or other algorithms, which are not limited here.

When the position and posture are calculated by the motion recovery structure (SFM) algorithm, the position and posture of the first imaging device when the first image is captured can be calculated by using the image control point as a constraint condition.

In a second possible manner, the first photographing device and the second photographing device are simultaneously photographed, and the ground station calculates the first photographing device based on a preset image processing algorithm (air triangulation algorithm, SFM algorithm or SLAM algorithm, etc.) The position and posture when the first image was taken. Calculating the second beat based on the relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated The position and posture of the camera when shooting the second image.

It should be noted that the position and posture obtained by the SLAM algorithm, the aerial triangulation algorithm or the SFM algorithm in the prior art are relative positions and relative postures in the shooting scene.

In order to make the position and posture obtained by the above calculations correspond to the world coordinate system, the position and posture corresponding to the image are more practically referenced. Alternatively, in this embodiment, the calculated relative position and relative posture can also be converted. Position and pose for the world coordinate system:

In a possible implementation manner, the ground station can use the GPS measuring device on the aircraft to acquire the GPS information of the aircraft. Specifically, the GPS information can be provided by the real-time dynamic control system (RTK), and the relative position obtained by the above calculation is The pose is converted to position and pose in the world coordinate system.

In another possible implementation manner, the ground station converts the position and posture obtained by the above calculation into the position and posture in the world coordinate system based on the GPS information of the preset image control point. Specifically, in this implementation, the relative position of the image control point in the first image captured by the first photographing device and the second image captured by the image capture point in the second photographing device may be manually found. The relative position, based on the relative position of the image control point and the GPS information of the image control point, converts the relative position and posture obtained by the above calculation into the position and posture in the world coordinate system. Or, by means of image recognition, firstly, based on the information of the GPS of the image control point, respectively searching for the image including the image control point from the first image captured by the first photographing device and the second image captured by the second photographing device, and There may be an area of the image control point in the image. Further, the image control point is identified in the above area by a preset machine learning model and an optimization algorithm, thereby obtaining the image control point in the first image and the second image. The relative position, further, based on the relative position of the image control points and the GPS information, converts the relative positions and postures of the first photographing device and the second photographing device when the image is captured into the position and posture in the world coordinate system. The above image recognition method improves the output image generation efficiency compared to the manual method.

Optionally, after determining the relative positions of the image control points in the first image and/or the second image, the embodiment may further display the relative positions of the image control points in the first image and/or the second image. To improve the user experience.

In yet another possible manner, the aircraft is at the first and second shooting devices The captured image is sent to the ground station and the GPS information of the aircraft is also sent to the ground station when the image is taken. The ground station converts the calculated relative position and posture into the position and posture in the world coordinates according to the GPS information corresponding to the image.

Optionally, when the first photographing device and the second photographing device are simultaneously photographed, the relative position and posture of the first photographing device when the first image is captured may be first converted into the world by any one of the above manners. The position and posture in the coordinate system, and further, based on the relative positional relationship between the first camera and the second camera, which are pre-calibrated, convert the relative position and posture of the second camera when the second image is captured, Position and attitude in the world coordinate system.

Step 103: Generate a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured.

For example, in the embodiment, the ground station performs dense matching according to the position and posture of the first photographing device when photographing the first image, and generates a corresponding dense point cloud or a semi-dense point cloud, and further, a point generated based on the dense matching. Clouds form and form a terrain surface. In the process of forming a terrain surface based on dense point matching and forming a terrain surface, the point cloud generated by the dense matching may be first divided into ground points and non-ground points, and then the ground points are extracted from the point cloud, based on the grounds. Point fitting forms the terrain surface. Of course, this is merely an example, and is not a limitation of the present invention. For example, in a real scene, a digital surface model (DSM) pre-stored in a ground station may be used as a terrain surface, and then the first photographing device may be photographed. The first image and the second image captured by the second photographing device are projected onto the DSM.

Optionally, when generating the terrain surface, a more accurate terrain surface can be calculated by using a preset image control point as a constraint.

Step 104: Perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.

In this embodiment, the first image and the second image may be projected based on the relative position and posture of the first photographing device when the first image is captured, and the relative position and posture of the second photographing device when the second image is captured. The position of the terrain surface may be based on the position and posture of the first photographing device in the world coordinate system when the first image is captured, and the position and posture of the second photographing device in the world coordinate system when the second image is captured. Will first image and second The image is projected onto the surface of the above terrain.

Optionally, the splicing method of the projection of the first image and the second image in the embodiment may be one of the following methods: a direct overlay method, a panoramic image splicing method, and each region of the final image is selected from the image center. Image methods, and stitching methods based on cost functions. Optionally, the second image projection may also be spliced based on the splicing line when the first image is projected and spliced. In this embodiment, a splicing method based on a cost function is taken as an example, and the splicing method is as follows:

In a possible splicing method, the ground station may first perform dense matching based on the position and posture of the first photographing device when taking the first image to generate a dense point cloud or a semi-dense point cloud, based on the second image on the terrain surface. The projection on the top, and the point cloud generated above, construct a cost function, and splicing the projection of the second image based on the cost function, so that the color difference on both sides of the splicing line is minimized. The splicing method of the first image projection is similar to the second image and will not be described here.

In another possible splicing method, the ground station may first construct a cost function based on the projection of the second image on the surface and the point cloud obtained based on the second image, and then perform the projection of the second image based on the cost function. splice.

Of course, the above splicing method is only an example, and is not a limitation on the present invention. In fact, any other splicing method may be adopted in an actual scenario.

Optionally, in this embodiment, the ground station can process the received image by using the following two working modes:

In one possible approach, the ground station processes the received image in a ready-to-go process. That is to say, when the ground station is in the cruising of the aircraft, the received image is processed to obtain a semi-dense point cloud or a dense point cloud of the image. In this way, the ground station updates the semi-dense point cloud, dense point cloud or sparse point cloud obtained by the processing for each image received. In addition, it should be noted that the above-mentioned processing method is not only the processing method included in the literal meaning, but depends on the processing speed of the ground station. If the processing speed of the ground station can support the reception or processing, then The ground station processes the image immediately after receiving the image. If the processing speed of the ground station is insufficient to support the instant processing of the image, the ground station sequentially processes the received image. Specifically, the ground station can follow the image. The receiving sequence is processed, and may be processed according to the storage order of the images, and may also be processed according to other customized processing sequences, and is not performed in this embodiment. Specifically limited.

In another possible processing manner, when the ground station is in the cruising of the aircraft, only the images captured by the first photographing device and the second photographing device are received, and when the aircraft ends the cruise photographing, the received images are collectively processed.

Optionally, when splicing the image projection, the global color adjustment and/or brightness adjustment of the calculated point cloud may be first performed to achieve the purpose of significantly improving image quality, and further, based on the adjusted Projecting the image, constructing the cost function by using the distance from the projected pixel to the photographing device as a constraint, and stitching the projection of the image onto the surface of the terrain based on the cost function, so that the color difference on both sides of the stitching line is minimized, so that the integrity can be obtained. Better output image.

Further, in order to avoid the influence of the non-ground point splicing in the point cloud (the non-ground point may cause the splicing misalignment), in the embodiment, when constructing the cost function, the non-ground point in the point cloud may also be excluded, so that the splicing is performed. The line can automatically avoid non-ground areas, resulting in a better visual output image.

Specifically, FIG. 3a and FIG. 3b are schematic diagrams of output images of two identical scenes provided by the present invention, wherein FIG. 3a is an output image obtained by using an estimated elevation surface as a projection surface, and FIG. 3b is formed by a point cloud fitting. The output image obtained by the terrain surface as the projection surface is spliced by the cost function method. As shown in Fig. 3a, in Fig. 3a, since the average elevation surface is used as the projection surface, the output image of Fig. 3a produces a severe stitching misalignment because the elevation surface cannot accurately fit the terrain surface. In Fig. 3b, because the terrain surface formed by the point cloud fitting is used as the projection surface, the terrain surface can be fitted more accurately, and the cost function can be used to minimize the chromatic aberration on both sides of the splicing line, so the output is obtained. There is no obvious dislocation phenomenon in the image, and the overall output image is better overall. Therefore, in the embodiment of the present invention, the terrain surface is fitted by the point cloud, and the cost function is used to perform the splicing processing, which can solve the problem of the output image mosaic misalignment.

Optionally, in order to make the entire output image have a better visual effect, after the first image and the second image are projected onto the terrain surface, the first image and the first image on the surface of the terrain may also be based on a preset strategy. / or projection of the second image for color and brightness adjustment. This enables a better stitching effect in the subsequent stitching process.

For example, FIG. 4a and FIG. 4b are outputs of two identical scenarios according to an embodiment of the present invention. In the image diagram, the projection on the terrain surface in Figure 4a is not processed by color and brightness. Therefore, the overall output image in Figure 4a is not very good in color and brightness, and the visual effect is poor, but in Figure 4b Because the brightness and color of the projection on the surface of the terrain are processed before the splicing, the obtained output image has better overall color and brightness, and the visual effect is better. Therefore, the embodiment of the present invention can effectively improve the visual effect of the output image by performing color and brightness processing on the projection on the terrain surface before the splicing process.

Optionally, the output image involved in this embodiment may be specifically an orthophoto, such as an orthophoto map or other image with real geographic coordinate information obtained according to orthographic projection.

In this embodiment, the first image obtained by the first photographing device with the FOV of the aircraft being greater than or equal to the preset threshold is acquired, and the second image obtained by the second photographing device carried by the aircraft is obtained, and is calculated based on a preset algorithm. a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when the second image is captured; based on the position and posture of the first image and the first photographing device when the first image is photographed, Generating a terrain surface; and based on the position and posture of the second photographing device when the second image is captured, the second image is projected and stitched on the surface of the terrain to obtain an output image. Since the FOV of the first photographing device is greater than or equal to the preset threshold in the embodiment, and the FOV is larger, the accuracy of the elevation surface (ie, the terrain surface) obtained by the image fitting based on the image captured by the first photographing device is higher, thereby The images captured by other shooting devices on the aircraft are projected onto the elevation surface to obtain a correspondingly accurate orthophoto image, which improves the accuracy of generating orthophotos.

FIG. 5 is a flowchart of a method for generating an output image according to an embodiment of the present invention. As shown in FIG. 5, based on the embodiment of FIG. 1, the method includes:

Step 501: Acquire a first visible light image obtained by the first photographing device mounted on the aircraft, and acquire a second visible light image obtained by the second photographing device mounted on the aircraft, where the first photographing device and the second photographing device are synchronized. Shooting.

In this embodiment, the first photographing device may be specifically a wide-angle camera, and the second photographing device is specifically a telephoto camera.

Step 502: Calculate a position and a posture of the first photographing device when the first visible light image is captured based on a preset image processing algorithm.

Step 503: Calculate a position and a posture of the second photographing device when the second visible light image is captured, based on a relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated.

Step 504: Generate a terrain surface based on the position and posture of the first visible light image and the first photographing device when the first visible light image is captured.

Step 505: Perform projection and splicing processing on the second visible light image on the terrain surface based on the position and posture of the second photographing device when the second visible light image is captured, to obtain a corresponding corresponding to the second photographing device. Visible light output image.

Optionally, in this embodiment, the method for splicing the projection of the second visible light image on the surface of the terrain includes:

In a possible implementation manner, the projection of the second visible light image on the terrain surface may be spliced based on the splicing line used for splicing the projection of the first visible light image on the terrain surface, and the second shooting device is obtained. Visible light output image.

In another possible implementation manner, a dense matching may be performed based on a position and a posture of the first photographing device when the first visible light image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; and then based on the second visible light image. Projecting on the surface of the terrain, and the point cloud generated by the dense matching, constructing a cost function; and splicing the projection of the second visible image on the terrain surface based on the cost function to obtain a visible light output image corresponding to the second imaging device .

In another possible implementation manner, a dense matching may be performed based on a position and a posture of the second photographing device when the second visible light image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; and then based on the second visible light image Projecting on the surface of the terrain, and the point cloud generated by the dense matching, constructing a cost function, splicing the projection of the second visible image on the surface of the terrain based on the cost function, and obtaining visible light corresponding to the second photographing device Output image.

Optionally, in this embodiment, the first visible light image is projected onto the terrain surface based on the position and posture of the first imaging device when the first visible light image is captured, and the projection based on the second visible light image on the terrain surface is further Performing image correction processing on the projection of the first visible light image on the first terrain surface, thereby obtaining a visible light output image corresponding to the first photographing device based on the image corrected processing projection, and based on the stitching line on the visible light output image, Splicing the projection of the second image on the surface of the terrain.

Optionally, in this embodiment, the orthographic visible light output image may be obtained by orthographic processing the projection of the second visible light image on the terrain surface.

In this embodiment, the first photographing device is specifically a wide-angle camera, and the second photographing device is specifically a telephoto camera, the FOV of the first photographing device is larger, and the FOV of the second photographing device is smaller, therefore, based on the first The position and posture of the shooting device when capturing the first visible light image, the accuracy and accuracy of the obtained elevation surface (ie, the terrain surface) is higher than the elevation obtained based on the position and posture of the second imaging device when capturing the second visible light image. Therefore, the second visible light image obtained by the second photographing device is projected onto the elevation surface to obtain an orthophoto image with high accuracy and precision, and the accuracy of the orthophoto image obtained by the second photographing device is improved.

The method and the beneficial effects of the method provided in this embodiment are similar to those in the embodiment of FIG. 1, and are not described herein again.

FIG. 6 is a flowchart of a method for generating an output image according to an embodiment of the present invention. As shown in FIG. 6 , on the basis of the embodiment of FIG. 1 , the method includes:

Step 601: Acquire a visible light image obtained by the first photographing device mounted on the aircraft, and a near-infrared image captured by the second photographing device, and the first photographing device and the first photographing device simultaneously photograph.

In this embodiment, the first photographing device may be specifically a visible light camera with a FOV greater than or equal to a preset threshold (such as a wide-angle camera with an FOV greater than or equal to a preset threshold), and the second photographing device may be specifically a near-infrared camera.

Step 602: Calculate a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm.

Step 603: Calculate a position and a posture of the second photographing device when photographing the near-infrared image based on a relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated.

Step 604: Generate a terrain surface based on the visible light image and the position and posture of the first photographing device when the visible light image is captured.

Step 605: Perform projection and splicing processing on the near-infrared image on the terrain surface based on a position and a posture of the second photographing device when the near-infrared image is captured. The near-infrared output image corresponding to the second photographing device.

Optionally, in this embodiment, the method for splicing the projection of the near-infrared image on the surface of the terrain includes:

In a possible implementation manner, the ground station can splicing the projection of the visible light image on the terrain surface to obtain an orthographic visible light output image, and based on the splicing line of the visible light image on the terrain surface, the near infrared The projection of the image on the surface of the terrain is spliced to obtain a near-infrared output image.

Optionally, the visible light output image corresponding to the first photographing device and the near infrared output image corresponding to the second photographing device are obtained, and the visible light output image and/or the near infrared output image may be displayed on the ground station in this embodiment. To enhance the user experience.

In another possible implementation manner, the position and posture of the first photographing device when capturing the visible light image may be densely matched to generate a corresponding dense point cloud or a semi-dense point cloud; and then photographed based on the second photographing device. The projection of the near-infrared image on the surface of the terrain and the point cloud generated by the dense matching described above construct a cost function; and based on the cost function, the projection of the near-infrared image on the terrain surface is spliced, and the corresponding corresponding to the second photographing device is obtained. Infrared output image.

In another possible implementation manner, the ground station performs dense matching based on the position and posture of the second photographing device when capturing the near-infrared image, and generates a corresponding dense point cloud or a semi-dense point cloud, and based on the The projection of the near-infrared image on the surface of the terrain, and the point cloud generated by the dense matching, construct a cost function, and the principle of constructing the cost function is to select a place where the color difference is small as much as possible, and avoid artificial building and the like. The feature, further, is based on the cost function to splicing the projection of the near-infrared image on the surface of the terrain to obtain an orthographic near-infrared output image.

Optionally, the embodiment further calculates the vegetation coverage index (NDVI) and/or the strong vegetation index (EVI) based on the visible light output image and the near-infrared output image obtained above, and obtains the NDVI and/or EVI based on the calculation. , draw the corresponding index map, and display the index map.

Further, after obtaining the index map, the analysis result may also analyze the growth state of the vegetation based on the index map, and output the analysis result. The purpose of providing vegetation analysis data is thus provided, which facilitates vegetation analysis.

In addition, due to the high elevation of trees and buildings in some scenarios, when classifying point clouds of near-infrared images, it is easy to divide trees into buildings or divide buildings into trees. The problem is that the NDVI and/or EVI obtained by the calculation can distinguish the vegetation and the building on the near-infrared image, and enhance the reliability of the classification.

For example, FIG. 7a is an unspliced near-infrared image obtained by a near-infrared camera, FIG. 7b is a near-infrared output image obtained by ortho-splicing of the near-infrared image shown in FIG. 7a, and FIG. 7c is a near-infrared image with FIG. 7a. The visible light output image corresponding to the visible light camera that is synchronously photographed by the camera, FIG. 7d is an NVDI index image calculated by using the red band image of the near-infrared image and the visible light output image, and FIG. 7e is a green image of the near-infrared image after the orthophoto stitching. Schematic diagram of the results of pseudo color rendering. As shown in Figures 7a-e, in Figure 7a, the coverage of a single NIR image is small, and the multiple unspliced images are not intuitive to the entire scene, but in Figure 7b, the The infrared output image not only better expresses the entire scene but also has measurable characteristics. The Fig. 7e obtained based on Fig. 7c - Fig. 7d can reflect the chlorophyll content of the plant by the brightness and saturation of the green, so that the analysis result is more intuitive.

In addition, it should be understood by those skilled in the art that although the present embodiment is an indicator for analyzing the vegetation growth status of NDVI and EVI index crops, the actual scene is not limited to the use of NDVI and EVI, but also NDVI and The EVI is replaced with other indicators that can be used to analyze the state of vegetation growth, and is not specifically limited in this embodiment.

In this embodiment, the visible light image and the near-infrared image are simultaneously acquired during image acquisition, and the NDVI and/or EVI index is calculated according to different responses of the plant to the two spectra, and the NDVI and/or EVI index is used as an important basis for classifying vegetation, and the NDVI and/or EVI index are used as an important basis for classifying vegetation. The reliability of point cloud classification.

Optionally, in this embodiment, the aircraft can also be equipped with a wide-angle camera, a telephoto camera, and a near-infrared camera, wherein the method for processing the image captured by the telephoto camera and the near-infrared camera based on the image captured by the wide-angle camera and the foregoing The embodiments are similar and will not be described again here.

The implementation manner and beneficial effects of the embodiment are similar to those of the embodiment of FIG. 1, and are not described herein again.

FIG. 8 is a flowchart of a method for generating an output image according to an embodiment of the present invention. As shown in FIG. 8 , based on the embodiment of FIG. 1 , the method includes:

Step 801: Acquire a visible light image obtained by the first photographing device mounted on the aircraft, and acquire an infrared image obtained by the second photographing device mounted on the aircraft, and the first photographing device and the second photographing device simultaneously photograph.

In this embodiment, the first photographing device may be specifically a visible light camera with a FOV greater than or equal to a preset threshold (such as a wide-angle camera with a FOV greater than or equal to a preset threshold), and the second photographing device may be specifically an infrared camera.

Step 802: Calculate a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm.

Step 803: Calculate a position and a posture of the second photographing device when the infrared image is captured based on a relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated.

Step 804: Generate a terrain surface based on the visible light image and the position and posture of the first photographing device when the visible light image is captured.

Step 805: Perform projection and splicing processing on the infrared image on the terrain surface to obtain an infrared output image corresponding to the second photographing device, based on the position and posture of the second photographing device when the infrared image is captured.

Optionally, in this embodiment, the method for splicing the projection of the infrared image on the surface of the terrain includes:

In a possible implementation manner, the ground station splicing the projection of the visible light image on the terrain surface to obtain a visible light output image, and stitching the projection of the infrared image based on the splicing line of the projection of the visible light image on the terrain surface. Obtain an infrared output image.

In another possible implementation manner, the position and posture of the first photographing device when capturing the visible light image may be densely matched to generate a corresponding dense point cloud or a semi-dense point cloud; and then photographed based on the second photographing device. The projection of the infrared image on the surface of the terrain and the point cloud generated by the dense matching described above construct a cost function; and based on the cost function, the projection of the infrared image on the terrain surface is spliced to obtain the infrared output image corresponding to the second imaging device. .

In another possible implementation manner, the ground station performs dense matching based on the position and posture of the second photographing device when capturing the infrared image, and generates a corresponding dense point cloud or semi-dense point cloud, and based on the infrared image in the terrain. Projection on the surface, and points generated by dense matching The cloud constructs a cost function and splicing the projection of the infrared image on the terrain surface based on the cost function to obtain an infrared output image.

Optionally, after obtaining the visible light output image and the infrared output image, the visible light output image and/or the infrared output image may be displayed on the ground station to improve the user experience.

Optionally, since the aircraft is equipped with an infrared camera in this embodiment, the embodiment can also identify the heat source object from the infrared image captured by the infrared camera or the infrared output image corresponding to the infrared camera based on the characteristics of the infrared image ( Such as photovoltaic panels, power lines, etc.). In particular, the power line is difficult to recognize on the visible light image of ordinary aerial photography because of its small diameter. However, the present embodiment can easily recognize the infrared image from the aerial image through the infrared image based on the characteristics of the power line heating. Identify the purpose of the power line.

Further, after the infrared image captured by the infrared camera or the infrared output image corresponding to the infrared camera identifies the power line, the embodiment can use the position and posture of the second imaging device when capturing the infrared image, and the preset power line mathematical model. The identified power line is modeled to form a power line layer, and the power line layer is superimposed and displayed on the visible light output image obtained above.

In this embodiment, based on the characteristic that the temperature of the power line is different from the temperature of the surrounding environment, the infrared line is mounted on the aircraft, and the infrared line image obtained by the infrared camera or the infrared output image corresponding to the infrared camera is used to identify the power line, and then the modeling method is adopted. The identified power line is modeled, a power line layer is generated, and the power line map is layered on the visible light output image, thereby realizing clear display of the power line on the orthophoto, and obtaining specific information of the power line by measurement. In addition, it is also possible to increase the information content of the orthophoto by displaying the specific parameters of the power line on the orthophoto.

Optionally, in this embodiment, the aircraft can also be equipped with a wide-angle camera, a telephoto camera, and an infrared camera at the same time, wherein the method for processing the image captured by the telephoto camera and the infrared camera based on the image captured by the wide-angle camera and the foregoing embodiment Similar, no longer repeat here.

FIG. 9 is a flowchart of a method for generating an output image according to an embodiment of the present invention, as shown in FIG. It is shown that, based on the embodiment of Figure 1, the method comprises:

Step 901: Acquire a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or equal to a preset threshold. Wherein the photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground.

Optionally, as shown in FIG. 10a, when the aircraft is flying at a fixed relative height with respect to the ground surface, the first photographing device and the second photographing device perform photographing at the same photographing interval in the horizontal direction.

Optionally, when the aircraft changes in height relative to the surface, the shooting interval of the first photographing device and the second photographing device changes. Specifically, as shown in FIG. 10b, when the aircraft is flying at a uniform absolute height, the first photographing device and the second photographing device are photographed in a horizontal direction at time-lapse shooting intervals, wherein The photographing interval is associated with a pre-configured image overlap rate and the relative height of the aircraft to the surface.

Optionally, in order to ensure the overlap ratio, the shooting interval may be correspondingly increased when the height is increased relative to the ground surface, and the shooting interval is correspondingly reduced when the height is reduced relative to the ground surface.

Step 902: Calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when the second image is photographed.

Step 903: Generate a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured.

Step 904: Perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.

The method and the beneficial effects of the method provided by this embodiment are similar to those of the embodiment of FIG. 1 and are not described herein again.

The embodiment of the invention provides a ground station, which may be the ground station described in the above embodiment. 11 is a schematic structural diagram of a ground station according to an embodiment of the present invention. As shown in FIG. 11, the ground station 10 includes: a communication interface 11, one or more processors 12; and one or more processors work independently or in cooperation. The interface 11 is connected to the processor 12; the communication interface 11 And configured to: obtain a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or equal to a preset threshold. The processor 12 is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured and a position of the second photographing device when the second image is photographed And the processor 12 is configured to: generate a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured; the processor 12 is further configured to: And based on the position and posture of the second photographing device when the second image is captured, the second image is projected and stitched on the terrain surface to obtain an output image.

Optionally, the FOV of the first photographing device is greater than the FOV of the second photographing device.

Optionally, the FOV of the second photographing device is less than the preset threshold.

Optionally, the processor 12 is configured to: determine, according to GPS information of a preset image control point, a position and a posture of the first photographing device when the first image is captured, and the second photographing The position and posture of the device when the second image is captured are converted into a position and a posture in the world coordinate system.

Optionally, the processor 12 is configured to: determine, according to GPS information of a preset image control point, a relative position of the image control point in the first image captured by the first photographing device; Determining a relative position of the image control point in the first image, and GPS information of the image control point, converting the position and posture of the first photographing device when the first image is captured into a world coordinate system Position and posture; converting the position and posture of the second photographing device when photographing the second image into world coordinates based on a relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated Position and posture under the system.

Optionally, the ground station further includes a display component 13 communicatively coupled to the processor 12, the display component 13 configured to: display the relative position of the image control point on the first image.

Optionally, the processor 12 is configured to: calculate, by using a motion recovery structure SFM algorithm, a position and a posture of the first photographing device when the first image is captured, by using a preset image control point as a constraint condition; And calculating a position and a posture of the second photographing device when the second image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.

Optionally, the processor 12 is configured to perform a dense matching based on a position and a posture of the first photographing device when the first image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; The resulting point cloud fits to form a terrain surface.

Optionally, the processor 12 is configured to: extract a ground point from a point cloud generated by the dense matching;

A terrain surface is formed based on the extracted ground point fit.

Optionally, the processor 12 further includes: performing global color and/or brightness adjustment on a projection of the second image on the surface.

Optionally, the preset image processing algorithm includes any one of the following: an aerial triangulation, an algorithm for recovering a structure SFM from motion, a real-time positioning, and a map construction SLAM algorithm.

Optionally, the output image includes an orthophoto.

Optionally, the shooting interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft relative to the ground.

Optionally, when the aircraft is flying at a fixed relative height with respect to the ground surface, the first photographing device and the second photographing device respectively photograph at the same photographing interval in the horizontal direction.

Optionally, when the aircraft changes in height relative to the surface, the shooting interval of the first photographing device and the second photographing device changes.

Optionally, when the aircraft is flying at a uniform absolute height, the first photographing device and the second photographing device are photographed in a horizontal direction at time-lapse photographing intervals, wherein the photographing interval is in advance The configured image overlap rate and the relative height of the aircraft to the surface.

The ground station provided in this embodiment can perform the technical solution of the embodiment of FIG. 1 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide a ground station. The ground station is based on the embodiment of FIG. 11 , and the communication interface 11 is configured to: acquire a first visible light image obtained by the first imaging device mounted on the aircraft, and acquire a second captured by the second imaging device mounted on the aircraft; The visible light image, the first photographing device and the second photographing device are simultaneously photographed. The processor 12 is configured to: calculate, according to a preset image processing algorithm, that the first photographing device is photographing the first a position and a posture of the visible light image; calculating a position and a posture of the second photographing device when the second visible light image is captured based on a relative positional relationship between the first photographing device and the second photographing device .

Optionally, the processor 12 is configured to: perform a projection on the terrain surface of the second visible light image based on a splicing line used when splicing the projection of the first visible light image on the terrain surface Splicing, obtaining a visible light output image corresponding to the second photographing device.

Optionally, the processor 12 is configured to perform a dense matching based on a position and a posture of the first photographing device when the first visible light image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the second visible light image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; stitching the projection of the second visible light image on the terrain surface based on the cost function Obtaining a visible light output image corresponding to the second photographing device.

Optionally, the processor 12 is configured to perform a dense matching based on a position and a posture of the second photographing device when the second visible light image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the second visible light image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; stitching the projection of the second visible light image on the terrain surface based on the cost function Obtaining a visible light output image corresponding to the second photographing device.

Optionally, the processor 12 is further configured to: project the first visible light image onto the terrain surface based on a position and a posture of the first photographing device when the first visible light image is captured.

Optionally, the processor 12 is further configured to orthographically process the projection of the second visible light image on the terrain surface.

Optionally, the first photographing device is a wide-angle camera, and the second photographing device is a telephoto camera.

The ground station provided by this embodiment can be used to perform the method of the embodiment of FIG. 5, and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide a ground station. The ground station is based on the embodiment of Figure 11, The communication interface 11 is configured to: acquire a visible light image captured by a first photographing device carried by an aircraft, and obtain a near-infrared image obtained by the second photographing device, the first photographing device and the first photographing The device shoots simultaneously. The processor 12 is configured to calculate, according to a preset image processing algorithm, a position and a posture of the first photographing device when capturing the visible light image; and the first photographing device and the second based on pre-calibration The relative positional relationship between the photographing devices is used to calculate the position and posture of the second photographing device when the near-infrared image is captured.

Optionally, the processor 12 is configured to: perform a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image; and perform splicing based on the projection of the visible light image on the surface of the terrain The stitching line is used to splicing the projection of the near-infrared image on the surface of the terrain to obtain a near-infrared output image.

Optionally, the display component 13 is configured to: display the visible light output image and/or the near infrared output image.

Optionally, the processor 13 is further configured to: calculate a vegetation coverage index NDVI and/or a strong vegetation index EVI based on the visible light output image and the near-infrared output image, and calculate the obtained NDVI and/or EVI, draw the corresponding index map.

Optionally, the display component 13 is further configured to: display the index map.

Optionally, the processor 13 is further configured to: analyze the growth status of the vegetation based on the index map, and output the analysis result.

Optionally, the processor 12 is configured to perform a dense matching based on a position and a posture of the first photographing device when capturing the visible light image, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the infrared image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; splicing the projection of the near-infrared image on the surface of the terrain based on the cost function to obtain a near-infrared Output image.

Optionally, the processor 12 is configured to perform a dense matching based on a position and a posture of the second photographing device when the near-infrared image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a near-infrared image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; stitching the projection of the near-infrared image on the surface of the terrain based on the cost function to obtain a near Infrared output image.

Optionally, the first photographing device is a wide-angle camera, and the second photographing device is near red Outside camera.

The ground station provided in this embodiment can be used to perform the method in the embodiment of FIG. 6, and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide a ground station. The ground station is based on the embodiment of FIG. 11 , and the communication interface 11 is configured to: acquire a visible light image obtained by the first imaging device mounted on the aircraft, and acquire an infrared image obtained by the second imaging device mounted on the aircraft. The first photographing device and the second photographing device are simultaneously photographed. The processor 12 is configured to calculate, according to a preset image processing algorithm, a position and a posture of the first photographing device when capturing the visible light image; and the first photographing device and the second based on pre-calibration The relative positional relationship between the photographing devices is used to calculate the position and posture of the second photographing device when the infrared image is captured.

Optionally, the processor 12 is configured to: perform splicing processing on the projection of the visible light image on the terrain surface to obtain a visible light output image; and perform splicing based on the projection of the visible light image on the terrain surface The stitching line used is used to splicing the projection of the infrared image to obtain an infrared output image.

Optionally, the display component 13 is configured to: display the infrared output image and/or the visible light output image.

Optionally, the processor 12 is configured to: identify a location of the heat source object in the infrared image captured by the second photographing device or the infrared output image.

Optionally, the display component 13 is further configured to: display a position of the heat source object in the infrared image or the infrared output image.

Optionally, the heat source object comprises a power line.

Optionally, the processor 12 is configured to: according to the position and posture of the second photographing device when capturing the infrared image, and the preset power line mathematical model, model the identified power line to form a power line diagram The display component 13 is configured to: superimpose and display the power line layer on the visible light output image.

Optionally, the processor 12 is configured to: perform dense matching based on a position and a posture of the first photographing device when capturing the visible light image, to generate a corresponding dense point cloud or a semi-dense point cloud; Projection of the image on the surface of the terrain, and the dense matching Generating a point cloud to construct a cost function; splicing the projection of the infrared image on the surface of the terrain based on the cost function to obtain an infrared output image.

Optionally, the processor 12 is configured to perform a dense matching based on a position and a posture of the second photographing device when the infrared image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; and stitching the projection of the infrared image on the surface of the terrain based on the cost function to obtain an infrared output image.

Optionally, the first photographing device is a wide-angle camera, and the second photographing device is an infrared camera.

The ground station provided by this embodiment can be used to perform the method of the embodiment of FIG. 8 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide a ground station. FIG. 12 is a schematic structural diagram of a ground station according to an embodiment of the present invention. As shown in FIG. 12, the ground station 20 includes: a communication interface 21, one or more processors 22; and the one or more processors 22 are separately or cooperatively Working, the communication interface 21 is connected to the processor 22; the communication interface 21 is configured to: acquire a first image captured by a first photographing device mounted on the aircraft, and acquire a second photographing device mounted on the aircraft. a second image, wherein an FOV of the first photographing device is greater than or equal to a preset threshold, wherein a photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft relative to the ground The processor 22 is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured and a position of the second photographing device when the second image is photographed And a gesture; the processor 22 is configured to: generate a terrain surface based on a position and a posture of the first image and the first photographing device when the first image is captured; 22: for performing projection and splicing processing on the second image on the terrain surface based on a position and a posture of the second photographing device when the second image is captured, to obtain an output image.

Optionally, when the aircraft is flying at a fixed relative height with respect to the ground surface, the first photographing device and the second photographing device perform photographing at the same photographing interval in the horizontal direction.

Optionally, when the aircraft changes in height relative to the surface, the first photographing device The shooting interval of the second photographing device changes.

The ground station provided by this embodiment can be used to perform the method of the embodiment of FIG. 9 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

An embodiment of the present invention provides an aircraft controller, which may be the aircraft controller described in the above embodiments. 13 is a schematic structural diagram of an aircraft controller according to an embodiment of the present invention. As shown in FIG. 13, the aircraft controller 30 includes: a communication interface 31, one or more processors 32; and one or more processors work alone or in cooperation. The communication interface 31 is connected to the processor 32. The communication interface 31 is configured to: acquire a first image captured by a first camera mounted on the aircraft, and acquire a second image obtained by the second camera mounted on the aircraft, where The FOV of the first photographing device is greater than or equal to a preset threshold; the processor 32 is configured to: calculate a position and a posture and a posture of the first photographing device when the first image is captured based on a preset algorithm a position and a posture of the second photographing device when the second image is captured; the processor 32 is configured to: based on the position of the first image and the first photographing device when photographing the first image, a gesture, generating a terrain surface; the processor 32 is further configured to: perform a second position on the terrain surface based on a position and a posture of the second photographing device when the second image is captured The image is projected and stitched to obtain an output image.

Optionally, the processor 32 is configured to: position and posture of the first photographing device when the first image is captured, and the second photographing based on GPS information of a preset image control point The position and posture of the device when the second image is captured are converted into a position and a posture in the world coordinate system.

Optionally, the processor 32 is configured to: determine, according to GPS information of a preset image control point, a relative position of the image control point in the first image captured by the first photographing device; Determining a relative position of the image control point in the first image, and the image control point GPS information, converting a position and a posture of the first photographing device when the first image is captured into a position and a posture in a world coordinate system; based on a pre-calibrated first photographing device and the second photographing device The relative positional relationship converts the position and posture of the second photographing device when the second image is captured into a position and a posture in the world coordinate system.

Optionally, the processor 32 is configured to: use a motion recovery structure SFM algorithm to calculate a position and a posture of the first photographing device when the first image is captured, by using a preset image control point as a constraint condition; And calculating a position and a posture of the second photographing device when the second image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.

Optionally, the processor 32 is configured to perform a dense matching based on a position and a posture of the first photographing device when the first image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; The resulting point cloud fits to form a terrain surface.

Optionally, the processor 32 is configured to: extract a ground point from a point cloud generated by the dense matching;

A terrain surface is formed based on the extracted ground point fit.

Optionally, the processor 32 further includes: performing global color and/or brightness adjustment on a projection of the second image on the surface.

Optionally, the output image includes an orthophoto.

Optionally, when the aircraft is flying at a uniform absolute height, the first photographing device and the second photographing device are photographed in a horizontal direction at time-lapse photographing intervals, wherein the photographing interval is in advance Configured image overlap rate, as well as the aircraft and the surface Relatively highly correlated.

The aircraft controller provided in this embodiment can perform the technical solution of the embodiment of FIG. 1 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide an aircraft controller. The aircraft controller is based on the embodiment of FIG. 13 , the communication interface 31 is configured to: acquire a first visible light image captured by a first imaging device mounted on the aircraft, and acquire a second captured image obtained by the second imaging device mounted on the aircraft The two visible light images are captured by the first photographing device and the second photographing device. The processor 32 is configured to calculate, according to a preset image processing algorithm, a position and a posture of the first photographing device when the first visible light image is captured; and the first photographing device and the second based on the pre-calibration A relative positional relationship of the photographing device is calculated, and a position and a posture of the second photographing device when the second visible light image is captured are calculated.

Optionally, the processor 32 is configured to: perform a projection on the terrain surface of the second visible light image based on a splicing line used when splicing the projection of the first visible light image on the terrain surface Splicing, obtaining a visible light output image corresponding to the second photographing device.

Optionally, the processor 32 is configured to perform a dense matching based on a position and a posture of the first photographing device when the first visible light image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the second visible light image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; stitching the projection of the second visible light image on the terrain surface based on the cost function Obtaining a visible light output image corresponding to the second photographing device.

Optionally, the processor 32 is configured to perform a dense matching based on a position and a posture of the second photographing device when the second visible light image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the second visible light image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; stitching the projection of the second visible light image on the terrain surface based on the cost function Obtaining a visible light output image corresponding to the second photographing device.

Optionally, the processor 32 is further configured to: project the first visible light image to the ground based on a position and a posture of the first photographing device when the first visible light image is captured. Shaped surface.

Optionally, the processor 32 is further configured to orthographically process the projection of the second visible light image on the terrain surface.

The aircraft controller provided in this embodiment can be used to perform the method of the embodiment of FIG. 5, and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide an aircraft controller. The aircraft controller is based on the embodiment of FIG. 13 , the communication interface 31 is configured to: acquire a visible light image obtained by a first photographing device mounted on an aircraft, and a near infrared image obtained by the second photographing device The first photographing device and the first photographing device are photographed simultaneously. The processor 32 is configured to calculate, according to a preset image processing algorithm, a position and a posture of the first photographing device when the visible light image is captured; and the first photographing device and the second The relative positional relationship between the photographing devices is used to calculate the position and posture of the second photographing device when the near-infrared image is captured.

Optionally, the processor 32 is configured to: perform splicing processing on the projection of the visible light image on the surface of the terrain to obtain a visible light output image; and perform splicing based on the projection of the visible light image on the terrain surface The stitching line is used to splicing the projection of the near-infrared image on the surface of the terrain to obtain a near-infrared output image.

Optionally, the processor 32 is further configured to: analyze the growth status of the vegetation based on the index map, and output the analysis result.

Optionally, the processor 32 is configured to perform a dense matching based on a position and a posture of the first photographing device when capturing the visible light image, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the infrared image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; splicing the projection of the near-infrared image on the surface of the terrain based on the cost function to obtain a near-infrared Output image.

Optionally, the processor 32 is configured to perform a dense matching based on a position and a posture of the second photographing device when the near-infrared image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a near-infrared image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; stitching the projection of the near-infrared image on the surface of the terrain based on the cost function to obtain a near Infrared output image.

Optionally, the first photographing device is a wide-angle camera, and the second photographing device is a near-infrared camera.

The aircraft controller provided by this embodiment can be used to perform the method of the embodiment of FIG. 6, and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide an aircraft controller. The aircraft controller is based on the embodiment of FIG. 13 , the communication interface 31 is configured to: acquire a visible light image captured by a first photographing device carried by the aircraft, and acquire an infrared image obtained by the second photographing device mounted on the aircraft, The first photographing device and the second photographing device are simultaneously photographed. The processor 32 is configured to calculate, according to a preset image processing algorithm, a position and a posture of the first photographing device when the visible light image is captured; and the first photographing device and the second The relative positional relationship between the photographing devices is used to calculate the position and posture of the second photographing device when the infrared image is captured.

Optionally, the processor 32 is configured to: perform splicing processing on the projection of the visible light image on the terrain surface to obtain a visible light output image; and perform splicing based on the projection of the visible light image on the terrain surface The stitching line used is used to splicing the projection of the infrared image to obtain an infrared output image.

Optionally, the processor 32 is configured to: identify a location of the heat source object in the infrared image captured by the second photographing device or the infrared output image.

Optionally, the heat source object comprises a power line.

Optionally, the processor 32 is configured to: according to the position and posture of the second photographing device when the infrared image is captured, and the preset power line mathematical model, model the identified power line to form a power line diagram The display component 13 is configured to: superimpose and display the power line layer on the visible light output image.

Optionally, the processor 32 is configured to: according to the first photographing device, photographing the The position and posture of the light image are densely matched to generate a corresponding dense point cloud or semi-dense point cloud; based on the projection of the infrared image on the surface of the terrain, and the point cloud generated by the dense matching, the cost is constructed a function; splicing a projection of the infrared image on the surface of the terrain based on the cost function to obtain an infrared output image.

Optionally, the processor 32 is configured to: perform dense matching based on a position and a posture of the second photographing device when the infrared image is captured, to generate a corresponding dense point cloud or a semi-dense point cloud; Projecting a projection of the image on the surface of the terrain, and a point cloud generated by the dense matching, constructing a cost function; and stitching the projection of the infrared image on the surface of the terrain based on the cost function to obtain an infrared output image.

The aircraft controller provided by this embodiment can be used to perform the method of the embodiment of FIG. 8 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide an aircraft controller. 14 is a schematic structural diagram of an aircraft controller according to an embodiment of the present invention. As shown in FIG. 14, the aircraft controller 40 includes: a communication interface 41, one or more processors 42; and the one or more processors 42 are separate Or cooperatively working, the communication interface 41 is connected to the processor 42; the communication interface 41 is configured to: acquire a first image captured by a first photographing device mounted on the aircraft, and acquire a second photographing device mounted on the aircraft. a second image obtained, wherein an FOV of the first photographing device is greater than or equal to a preset threshold, wherein a photographing interval of the first photographing device and the second photographing device and a flight of the aircraft relative to the ground The processor 42 is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured, and when the second photographing device captures the second image Position and posture; the processor 42 is configured to: generate a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured The processor 42 is configured to: perform projection and splicing processing on the second image on the terrain surface based on a position and a posture of the second photographing device when the second image is captured, to obtain an output image. .

Optionally, when the aircraft is flying at a fixed relative height with respect to the ground surface, the first photographing device and the second photographing device are performed at the same photographing interval in the horizontal direction. Shooting.

The aircraft controller provided in this embodiment can be used to perform the method of the embodiment of FIG. 9 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

The embodiment of the present invention provides a computer readable storage medium, including instructions, when executed on a computer, causing a computer to execute the output image generating method provided by the foregoing embodiment.

Embodiments of the present invention provide a drone. The drone includes a fuselage; a fuselage; a power system mounted on the body for providing flight power; and a first photographing device and a second photographing device mounted on the body for capturing images, wherein the drone; The FOV of the first photographing device is greater than or equal to a preset threshold; and the aircraft controller as described in the above embodiments.

The execution mode and the beneficial effects of the unmanned aerial vehicle provided in this embodiment are the same as those of the aircraft controller in the foregoing embodiment, and are not described herein again.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above-described integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The above software functional unit is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform the methods of the various embodiments of the present invention. Part of the steps. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes. .

A person skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of each functional module described above is exemplified. In practical applications, the above function assignment can be completed by different functional modules as needed, that is, the device is installed. The internal structure is divided into different functional modules to perform all or part of the functions described above. For the specific working process of the device described above, refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that The technical solutions described in the foregoing embodiments may be modified, or some or all of the technical features may be equivalently replaced; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present invention. range.

Claims

An output image generating method, comprising:

Obtaining a first image obtained by the first photographing device carried by the aircraft, and acquiring a second image obtained by the second photographing device mounted on the aircraft, wherein the first photographing device has a field of view angle FOV greater than or equal to a preset threshold ;

Calculating a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when photographing the second image, based on a preset algorithm;

Generating a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured;

And based on the position and posture of the second photographing device when the second image is captured, the second image is projected and stitched on the terrain surface to obtain an output image.
The method of claim 1 wherein the FOV of the first photographing device is greater than the FOV of the second photographing device.
The method of claim 2 wherein the FOV of the second photographing device is less than the predetermined threshold.
The method according to claim 3, wherein the acquiring the first image obtained by the first photographing device carried by the aircraft and acquiring the second image obtained by the second photographing device carried by the aircraft comprises:

Obtaining a first visible light image obtained by the first photographing device mounted on the aircraft, and acquiring a second visible light image obtained by the second photographing device mounted on the aircraft, the first photographing device and the second photographing device simultaneously photographing.
The method according to claim 4, wherein the calculating, based on a preset algorithm, a position and a posture of the first photographing device when the first image is captured and the second photographing device are photographing the photographing The position and posture of the second image, including:

Calculating a position and a posture of the first photographing device when the first visible light image is captured based on a preset image processing algorithm;

And calculating a position and a posture of the second photographing device when the second visible light image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The method of claim 4 wherein said surface of said terrain Performing splicing processing on the second image, including:

And splicing the projection of the second visible light image on the terrain surface according to a splicing line used for splicing the projection of the first visible light image on the surface of the terrain, and obtaining the corresponding second shooting device Visible light output image.
The method according to claim 4, wherein said splicing said second image on said surface of said terrain comprises:

Performing dense matching based on the position and posture of the first photographing device when photographing the first visible light image to generate a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the second visible light image on the terrain surface and a point cloud generated by the dense matching;

The projection of the second visible light image on the terrain surface is spliced based on the cost function, and the visible light output image corresponding to the second imaging device is obtained.
The method of claim 4, wherein the method further comprises:

The first visible light image is projected onto the terrain surface based on a position and a posture of the first photographing device when the first visible light image is captured.
The method according to claim 6 or 7, wherein the method further comprises:

Projecting the projection of the second visible light image on the surface of the terrain to orthographic processing.
The method according to any one of claims 1 to 9, wherein the first photographing device is a wide-angle camera and the second photographing device is a telephoto camera.
The method according to claim 1, wherein the acquiring the first image obtained by the first photographing device carried by the aircraft and acquiring the second image obtained by the second photographing device carried by the aircraft comprises:

Obtaining a visible light image obtained by the first photographing device mounted on the aircraft, and a near-infrared image obtained by the second photographing device, the first photographing device and the first photographing device simultaneously photographing.
The method according to claim 11, wherein the calculating, based on a preset algorithm, a position and a posture of the first photographing device when the first image is captured and the second photographing device are photographing the photographing The position and posture of the second image, including:

Calculating the visible by the first photographing device based on a preset image processing algorithm Position and posture of the light image;

And calculating a position and a posture of the second photographing device when photographing the near-infrared image based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The method according to claim 11, wherein said splicing processing said second image on said terrain surface comprises:

Performing a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image;

The projection of the near-infrared image on the surface of the terrain is spliced based on a splicing line used for splicing the projection of the visible light image on the surface of the terrain to obtain a near-infrared output image.
The method of claim 13 wherein the method further comprises:

Displaying the visible light output image and/or the near infrared output image.
The method of claim 13 wherein the method further comprises:

Calculating a vegetation coverage index NDVI and/or a strong vegetation index EVI based on the visible light output image and the near-infrared output image, and drawing a corresponding index map based on the calculated NDVI and/or EVI.
The method of claim 15 wherein the method further comprises:

The index map is displayed.
The method of claim 16 wherein the method further comprises:

Based on the index map, the growth state of the vegetation is analyzed, and the analysis result is output.
The method according to claim 11, wherein said splicing processing said second image on said terrain surface comprises:

Performing dense matching based on the position and posture of the first photographing device when capturing the visible light image, and generating a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the near-infrared image on the surface of the terrain and a point cloud generated by the dense matching;

A projection of the near-infrared image on the surface of the terrain is spliced based on the cost function to obtain a near-infrared output image.
Method according to any of claims 11-18, wherein said One camera is a wide-angle camera and the second camera is a near-infrared camera.
The method according to claim 1, wherein the acquiring the first image obtained by the first photographing device carried by the aircraft and acquiring the second image obtained by the second photographing device carried by the aircraft comprises:

Obtaining a visible light image obtained by the first photographing device mounted on the aircraft, and acquiring an infrared image obtained by the second photographing device mounted on the aircraft, the first photographing device and the second photographing device simultaneously photographing.
The method according to claim 20, wherein the calculating, based on a preset algorithm, a position and a posture of the first photographing device when the first image is captured and the second photographing device are photographing the photographing The position and posture of the second image, including:

Calculating a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm;

And calculating a position and a posture of the second photographing device when the infrared image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The method according to claim 20, wherein said splicing said second image on said terrain surface comprises:

Performing a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image;

And projecting the projection of the infrared image based on a splicing line used for splicing the projection of the visible light image on the surface of the terrain to obtain an infrared output image.
The method of claim 22, further comprising: displaying the infrared output image and/or the visible light output image.
The method of claim 22, wherein the method further comprises:

A position of the heat source object is recognized in the infrared image obtained by the second photographing device or the infrared output image.
The method of claim 24, wherein the method further comprises:

A position of the heat source object in the infrared image or the infrared output image is displayed.
The method of claim 24 wherein said heat source object comprises a power line.
The method of claim 26, wherein the method further comprises:

Determining the identified power line based on the position and posture of the second photographing device when the infrared image is captured, and the preset power line mathematical model to form a power line layer;

The power line layer is superimposed on the visible light output image.
The method according to claim 20, wherein said splicing said second image on said terrain surface comprises:

Performing dense matching based on the position and posture of the first photographing device when capturing the visible light image, and generating a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the infrared image on the surface of the terrain and a point cloud generated by the dense matching;

The projection of the infrared image on the surface of the terrain is spliced based on the cost function to obtain an infrared output image.
The method according to any one of claims 20 to 28, wherein the first photographing device is a wide-angle camera and the second photographing device is an infrared camera.
The method according to claim 1, wherein the calculating, based on a preset algorithm, a position and a posture of the first photographing device when the first image is captured and the second photographing device are photographing the photographing After the position and posture of the second image, the method further includes:

Positioning and posture of the first photographing device when photographing the first image, and a position of the second photographing device when photographing the second image, based on GPS information of a preset image control point The pose is converted to the position and posture in the world coordinate system.
The method according to claim 30, wherein said position and posture of said first photographing device when photographing said first image is based on GPS information of a preset image control point, and said The position and posture of the second photographing device when the second image is captured is converted into a position and a posture in the world coordinate system, including:

Determining a relative position of the image control point in the first image captured by the first photographing device based on GPS information of a preset image control point;

Converting a position and a posture of the first photographing device when the first image is captured into a world coordinate based on a relative position of the image control point in the first image and GPS information of the image control point Position and posture;

Based on a relative position between the first photographing device and the second photographing device that are pre-calibrated The relationship is that the position and posture of the second photographing device when the second image is captured is converted into a position and a posture in the world coordinate system.
The method of claim 31, wherein the method further comprises:

Displaying the relative position of the image control point on the first image.
The method according to claim 1, wherein the calculating, based on a preset algorithm, a position and a posture of the first photographing device when the first image is captured and the second photographing device are photographing the photographing The position and posture of the second image, including:

Calculating a position and a posture of the first photographing device when the first image is captured by using a motion recovery structure SFM algorithm with a preset image control point as a constraint condition;

And calculating a position and a posture of the second photographing device when the second image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The method according to claim 1, wherein the generating a terrain surface based on the position and posture of the first photographing device when the first image is captured comprises:

Performing dense matching based on the position and posture of the first photographing device when photographing the first image, and generating a corresponding dense point cloud or semi-dense point cloud;

A point cloud fit based on dense matching forms a terrain surface.
The method according to claim 34, wherein the forming of the topographic surface by the point cloud based on the dense matching comprises:

Extracting ground points from a point cloud generated by dense matching;

A terrain surface is formed based on the extracted ground point fit.
The method of claim 1 further comprising:

Global color and/or brightness adjustment is performed on the projection of the second image on the surface.
The method according to claim 5 or 12 or 21, wherein the preset image processing algorithm comprises any one of the following: aerial triangulation, algorithm for recovering structure SFM from motion, real-time positioning and map construction SLAM algorithm .
The method of any of claims 1 to 37, wherein the output image comprises an orthophoto.
The method according to any one of claims 1 to 3, wherein the photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground.
The method according to claim 39, wherein said first photographing device and said second photographing device respectively photograph in the same direction in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface Shoot at intervals.
The method according to claim 39, wherein a photographing interval of said first photographing device and said second photographing device changes when said aircraft changes in height with respect to a surface.
The method according to claim 41, wherein said first photographing device and said second photographing device photograph in a horizontal direction at time-lapse photographing intervals when said aircraft is flying at a uniform absolute height Wherein the shooting interval is associated with a pre-configured image overlay rate and a relative height of the aircraft to the surface.
An output image generating method, comprising:

Acquiring a first image obtained by the first photographing device carried by the aircraft, and obtaining a second image obtained by the second photographing device mounted on the aircraft, wherein the FOV of the first photographing device is greater than or equal to a preset threshold, wherein a photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground;

Calculating a position and a posture of the first photographing device when the first image is captured and a position and a posture of the second photographing device when photographing the second image, based on a preset algorithm;

Generating a terrain surface based on the position and posture of the first image and the first photographing device when the first image is captured;

And based on the position and posture of the second photographing device when the second image is captured, the second image is projected and stitched on the terrain surface to obtain an output image.
The method according to claim 43, wherein said first photographing device and said second photographing device are at the same photographing interval in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface Take a picture.
The method according to claim 43, wherein a photographing interval of said first photographing device and said second photographing device changes when said aircraft changes in height with respect to a surface.
The method according to claim 45, wherein said first photographing device and said second photographing device photograph in a horizontal direction at time-lapse photographing intervals when said aircraft is flying at a uniform absolute height Where the shooting interval and the pre-configured shadow Like the overlap rate, and the relative height of the aircraft to the surface.
A ground station, comprising: a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or Equal to the preset threshold;

The processor is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured, and a position and a posture of the second photographing device when the second image is photographed;

The processor is configured to generate a terrain surface based on a position and a posture of the first image and the first photographing device when the first image is captured;

The processor is further configured to: perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.
The ground station according to claim 47, wherein the FOV of the first photographing device is greater than the FOV of the second photographing device.
The ground station of claim 48, wherein the FOC of the second photographing device is less than the predetermined threshold.
The ground station according to claim 49, wherein the communication interface is configured to: acquire a first visible light image obtained by the first photographing device mounted on the aircraft, and acquire a second photographed by the second photographing device mounted on the aircraft The two visible light images are captured by the first photographing device and the second photographing device.
The ground station according to claim 50, wherein said processor is configured to:

Calculating a position and a posture of the first photographing device when the first visible light image is captured based on a preset image processing algorithm;

And calculating a position and a posture of the second photographing device when the second visible light image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The ground station according to claim 50, wherein the processor is configured to: align a second visible light based on a splicing line used when splicing the projection of the first visible light image on the terrain surface The projection of the image on the surface of the terrain is spliced to obtain a visible light output image corresponding to the second photographing device.
The ground station of claim 50 wherein said processor is operative to:

Performing dense matching based on the position and posture of the first photographing device when photographing the first visible light image to generate a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the second visible light image on the terrain surface and a point cloud generated by the dense matching;

The projection of the second visible light image on the terrain surface is spliced based on the cost function, and the visible light output image corresponding to the second imaging device is obtained.
The ground station according to claim 50, wherein the processor is further configured to: display the first visible light image based on a position and a posture of the first photographing device when the first visible light image is captured Projected onto the surface of the terrain.
The ground station according to claim 52 or 53, wherein the processor is further configured to orthographically process the projection of the second visible light image on the terrain surface.
A ground station according to any one of claims 47 to 55, wherein the first photographing device is a wide-angle camera and the second photographing device is a telephoto camera.
The ground station according to claim 47, wherein the communication interface is configured to: acquire a visible light image obtained by a first photographing device mounted on the aircraft, and a near infrared image obtained by the second photographing device The first photographing device and the first photographing device are photographed simultaneously.
A ground station according to claim 57, wherein said processor is for:

Calculating a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm;

Calculating a position and a posture of the second photographing device when photographing the near-infrared image based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated state.
A ground station according to claim 57, wherein said processor is for:

Performing a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image;

The projection of the near-infrared image on the surface of the terrain is spliced based on a splicing line used for splicing the projection of the visible light image on the surface of the terrain to obtain a near-infrared output image.
A ground station according to claim 59, wherein said ground station further comprises: a display component, said display component being communicatively coupled to said processor;

The display component is configured to: display the visible light output image and/or the near infrared output image.
The ground station according to claim 59, wherein the processor is further configured to: calculate a vegetation coverage index NDVI and/or a strong vegetation index EVI based on the visible light output image and the near-infrared output image, And based on the calculated NDVI and / or EVI, draw the corresponding index map.
A ground station according to claim 61, wherein the display component is operative to: display the index map.
The ground station of claim 62, wherein the processor is further configured to:

Based on the index map, the growth state of the vegetation is analyzed, and the analysis result is output.
A ground station according to claim 57, wherein said processor is for:

Performing dense matching based on the position and posture of the first photographing device when capturing the visible light image, and generating a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the near-infrared image on the surface of the terrain and a point cloud generated by the dense matching;

A projection of the near-infrared image on the surface of the terrain is spliced based on the cost function to obtain a near-infrared output image.
A ground station according to any one of claims 57-64, wherein said said The first photographing device is a wide-angle camera and the second photographing device is a near-infrared camera.
The ground station according to claim 47, wherein the communication interface is configured to: acquire a visible light image obtained by the first photographing device mounted on the aircraft, and acquire an infrared image obtained by the second photographing device mounted on the aircraft, The first photographing device and the second photographing device are simultaneously photographed.
The ground station of claim 66, wherein said processor is configured to:

Calculating a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm;

And calculating a position and a posture of the second photographing device when the infrared image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The ground station of claim 66, wherein said processor is configured to:

Performing a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image;

And projecting the projection of the infrared image based on a splicing line used for splicing the projection of the visible light image on the surface of the terrain to obtain an infrared output image.
A ground station according to claim 68, wherein the display component is operative to: display the infrared output image and/or the visible light output image.
The ground station of claim 68 wherein said processor is operative to:

A position of the heat source object is recognized in the infrared image obtained by the second photographing device or the infrared output image.
The ground station of claim 70 wherein the display component is further configured to:

A position of the heat source object in the infrared image or the infrared output image is displayed.
A ground station according to claim 70, wherein said heat source object comprises a power line.
A ground station according to claim 72, wherein said processor is And: forming a power line layer based on the position and posture of the second photographing device when the infrared image is captured, and a preset power line mathematical model, and modeling the identified power line;

The display component is configured to: superimpose and display the power line layer on the visible light output image.
The ground station of claim 66, wherein said processor is configured to:

Performing dense matching based on the position and posture of the first photographing device when capturing the visible light image, and generating a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the infrared image on the surface of the terrain and a point cloud generated by the dense matching;

The projection of the infrared image on the surface of the terrain is spliced based on the cost function to obtain an infrared output image.
A ground station according to any one of claims 66-74, wherein the first photographing device is a wide-angle camera and the second photographing device is an infrared camera.
The ground station according to claim 47, wherein said processor is configured to:

Positioning and posture of the first photographing device when photographing the first image, and a position of the second photographing device when photographing the second image, based on GPS information of a preset image control point The pose is converted to the position and posture in the world coordinate system.
The ground station of claim 76 wherein said processor is operative to:

Determining a relative position of the image control point in the first image captured by the first photographing device based on GPS information of a preset image control point;

Converting a position and a posture of the first photographing device when the first image is captured into a world coordinate based on a relative position of the image control point in the first image and GPS information of the image control point Position and posture;

Converting the position and posture of the second photographing device when photographing the second image into a position in the world coordinate system and based on a relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated attitude.
A ground station according to claim 77, wherein the display component is for:

Displaying the relative position of the image control point on the first image.
The ground station according to claim 47, wherein said processor is configured to:

Calculating a position and a posture of the first photographing device when the first image is captured by using a motion recovery structure SFM algorithm with a preset image control point as a constraint condition;

And calculating a position and a posture of the second photographing device when the second image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The ground station according to claim 47, wherein said processor is configured to:

Performing dense matching based on the position and posture of the first photographing device when photographing the first image, and generating a corresponding dense point cloud or semi-dense point cloud;

A point cloud fit based on dense matching forms a terrain surface.
The ground station of claim 80 wherein said processor is operative to:

Extracting ground points from a point cloud generated by dense matching;

A terrain surface is formed based on the extracted ground point fit.
The ground station of claim 47, wherein the processor further comprises:

Global color and/or brightness adjustment is performed on the projection of the second image on the surface.
The ground station according to claim 51 or 58 or 67, wherein said preset image processing algorithm comprises any one of the following: aerial triangulation, algorithm for recovering structure SFM from motion, real-time positioning and map construction SLAM algorithm.
A ground station according to any of claims 47-83, wherein the output image comprises an orthophoto.
A ground station according to any one of claims 47-83, wherein the photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground.
The ground station according to claim 85, wherein said first photographing device and said second photographing device respectively have the same in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface Shooting at the shooting interval.
A ground station according to claim 85, wherein said aircraft phase The photographing interval of the first photographing device and the second photographing device changes when the surface height is changed.
A ground station according to claim 87, wherein said first photographing device and said second photographing device are operated in a horizontal direction at time-varying photographing intervals when said aircraft is flying at a uniform absolute height Shooting, wherein the shooting interval is associated with a pre-configured image overlap rate and a relative height of the aircraft to the surface.
A ground station, comprising: a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or Is equal to a preset threshold, wherein a photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground;

The processor is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured, and a position and a posture of the second photographing device when the second image is photographed;

The processor is configured to generate a terrain surface based on a position and a posture of the first image and the first photographing device when the first image is captured;

The processor is configured to: perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.
A ground station according to claim 89, wherein said first photographing device and said second photographing device are photographed in the same direction in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface. Shoot at intervals.
The ground station according to claim 89, wherein a photographing interval of said first photographing device and said second photographing device changes when said aircraft changes in height with respect to a surface.
The ground station according to claim 91, wherein said first photographing device and said second photographing device are horizontal when said aircraft is flying at a uniform absolute height The photographing is performed in a temporally varying photographing interval, wherein the photographing interval is associated with a pre-configured image overlap rate and a relative height of the aircraft to the surface.
An aircraft controller, comprising: a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire a first image obtained by the first photographing device mounted on the aircraft, and acquire a second image obtained by the second photographing device mounted on the aircraft, where the FOV of the first photographing device is greater than or Equal to the preset threshold;

The processor is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured, and a position and a posture of the second photographing device when the second image is photographed;

The processor is configured to generate a terrain surface based on a position and a posture of the first image and the first photographing device when the first image is captured;

The processor is further configured to: perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.
The aircraft controller of claim 93, wherein the FOV of the first photographing device is greater than the FOV of the second photographing device.
The aircraft controller of claim 94, wherein the FOC of the second photographing device is less than the predetermined threshold.
The aircraft controller according to claim 95, wherein the communication interface is configured to: acquire a first visible light image obtained by the first photographing device mounted on the aircraft, and acquire a second photographing device mounted on the aircraft. The second visible light image is captured by the first photographing device and the second photographing device.
The aircraft controller according to claim 96, wherein said processor is configured to:

Calculating a position and a posture of the first photographing device when the first visible light image is captured based on a preset image processing algorithm;

Calculating a position and a posture of the second photographing device when photographing the second visible light image based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated state.
The aircraft controller according to claim 96, wherein said processor is configured to: splicing a line based on a projection of said first visible light image on said terrain surface, said second The projection of the visible light image on the surface of the terrain is spliced to obtain a visible light output image corresponding to the second photographing device.
The aircraft controller of claim 96 wherein said processor is operative to:

Performing dense matching based on the position and posture of the first photographing device when photographing the first visible light image to generate a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the second visible light image on the terrain surface and a point cloud generated by the dense matching;

The projection of the second visible light image on the terrain surface is spliced based on the cost function, and the visible light output image corresponding to the second imaging device is obtained.
The aircraft controller according to claim 96, wherein the processor is further configured to: based on a position and a posture of the first photographing device when photographing the first visible light image, the first visible light The image is projected onto the surface of the terrain.
The aircraft controller according to claim 98 or claim 99, wherein the processor is further configured to orthographically process the projection of the second visible light image on the terrain surface.
The aircraft controller according to any one of claims 93-101, wherein the first photographing device is a wide-angle camera and the second photographing device is a telephoto camera.
The aircraft controller according to claim 93, wherein the communication interface is configured to: acquire a visible light image obtained by the first photographing device mounted on the aircraft, and a near infrared image obtained by the second photographing device Image, the first photographing device and the first photographing device are simultaneously photographed.
The aircraft controller of claim 103 wherein said processor is operative to:

Calculating a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm;

Relating between the first photographing device and the second photographing device based on pre-calibration a positional relationship, calculating a position and a posture of the second photographing device when photographing the near-infrared image.
The aircraft controller of claim 103 wherein said processor is operative to:

Performing a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image;

The projection of the near-infrared image on the surface of the terrain is spliced based on a splicing line used for splicing the projection of the visible light image on the surface of the terrain to obtain a near-infrared output image.
The aircraft controller according to claim 105, wherein the processor is further configured to: calculate a vegetation coverage index NDVI and/or a strong vegetation index EVI based on the visible light output image and the near infrared output image And based on the calculated NDVI and / or EVI, draw the corresponding index map.
The aircraft controller of claim 106, wherein the processor is further configured to:

Based on the index map, the growth state of the vegetation is analyzed, and the analysis result is output.
The aircraft controller of claim 103 wherein said processor is operative to:

Performing dense matching based on the position and posture of the first photographing device when capturing the visible light image, and generating a corresponding dense point cloud or semi-dense point cloud;

Constructing a cost function based on a projection of the near-infrared image on the surface of the terrain and a point cloud generated by the dense matching;

A projection of the near-infrared image on the surface of the terrain is spliced based on the cost function to obtain a near-infrared output image.
The aircraft controller according to any one of claims 103 to 108, wherein the first photographing device is a wide-angle camera and the second photographing device is a near-infrared camera.
The aircraft controller according to claim 93, wherein the communication interface is configured to: acquire a visible light image obtained by the first photographing device mounted on the aircraft, and acquire an infrared image obtained by the second photographing device mounted on the aircraft. , the first shot The device and the second photographing device are photographed simultaneously.
The aircraft controller of claim 110 wherein said processor is operative to:

Calculating a position and a posture of the first photographing device when the visible light image is captured based on a preset image processing algorithm;

And calculating a position and a posture of the second photographing device when the infrared image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The aircraft controller of claim 110 wherein said processor is operative to:

Performing a splicing process on the projection of the visible light image on the surface of the terrain to obtain a visible light output image;

And projecting the projection of the infrared image based on a splicing line used for splicing the projection of the visible light image on the surface of the terrain to obtain an infrared output image.
The aircraft controller of claim 112 wherein said processor is operative to:

A position of the heat source object is recognized in the infrared image obtained by the second photographing device or the infrared output image.
The aircraft controller of claim 113 wherein said heat source object comprises a power line.
The aircraft controller of claim 114, wherein the processor is configured to:

Determining the identified power line based on the position and posture of the second photographing device when the infrared image is captured, and the preset power line mathematical model to form a power line layer;

The power line layer is superimposed on the visible light output image.
The aircraft controller of claim 110 wherein said processor is operative to:

Performing dense matching based on the position and posture of the first photographing device when capturing the visible light image, and generating a corresponding dense point cloud or semi-dense point cloud;

Projecting a projection on the surface of the terrain based on the infrared image, and the dense matching a point cloud, constructing a cost function;

The projection of the infrared image on the surface of the terrain is spliced based on the cost function to obtain an infrared output image.
The aircraft controller according to any one of claims 110 to 116, wherein the first photographing device is a wide-angle camera and the second photographing device is an infrared camera.
The aircraft controller of claim 93 wherein said processor is operative to:

Positioning and posture of the first photographing device when photographing the first image, and a position of the second photographing device when photographing the second image, based on GPS information of a preset image control point The pose is converted to the position and posture in the world coordinate system.
The aircraft controller of claim 118 wherein said processor is operative to:

Determining a relative position of the image control point in the first image captured by the first photographing device based on GPS information of a preset image control point;

Converting a position and a posture of the first photographing device when the first image is captured into a world coordinate based on a relative position of the image control point in the first image and GPS information of the image control point Position and posture;

Converting the position and posture of the second photographing device when photographing the second image into a position in the world coordinate system and based on a relative positional relationship between the first photographing device and the second photographing device that are pre-calibrated attitude.
The aircraft controller of claim 93 wherein said processor is operative to:

Calculating a position and a posture of the first photographing device when the first image is captured by using a motion recovery structure SFM algorithm with a preset image control point as a constraint condition;

And calculating a position and a posture of the second photographing device when the second image is captured based on a relative positional relationship between the first photographing device and the second photographing device that is pre-calibrated.
The aircraft controller of claim 93 wherein said processor is operative to:

Performing dense matching based on the position and posture of the first photographing device when photographing the first image, and generating a corresponding dense point cloud or semi-dense point cloud;

A point cloud fit based on dense matching forms a terrain surface.
The aircraft controller of claim 121 wherein said processor is operative to:

Extracting ground points from a point cloud generated by dense matching;

A terrain surface is formed based on the extracted ground point fit.
The aircraft controller of claim 93, wherein the aircraft controller further comprises:

Global color and/or brightness adjustment is performed on the projection of the second image on the surface.
The aircraft controller according to claim 97 or 104 or 111, wherein the preset image processing algorithm comprises any one of the following: aerial triangulation, algorithm for recovering structure SFM from motion, real-time positioning and map construction SLAM algorithm.
The aircraft controller of any of claims 93-124, wherein the output image comprises an orthophoto.
The aircraft controller according to any one of claims 93-124, wherein the photographing interval of the first photographing device and the second photographing device is associated with a flying height of the aircraft with respect to the ground.
The aircraft controller according to claim 126, wherein said first photographing device and said second photographing device are respectively identical in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface The shooting interval is taken.
The aircraft controller according to claim 126, wherein a photographing interval of said first photographing device and said second photographing device changes when said aircraft changes in height with respect to a surface.
The aircraft controller according to claim 128, wherein said first photographing device and said second photographing device are time-variant photographing intervals in a horizontal direction when said aircraft is flying at a uniform absolute height Shooting is performed wherein the shooting interval is associated with a pre-configured image overlay rate and the relative height of the aircraft to the surface.
An aircraft controller, comprising: a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire a first photograph obtained by the first photographing device carried by the aircraft An image obtained by acquiring a second image obtained by the second photographing device mounted on the aircraft, wherein the FOV of the first photographing device is greater than or equal to a preset threshold, wherein the first photographing device and the second photographing The shooting interval of the device is associated with the flying height of the aircraft relative to the ground;

The processor is configured to calculate, according to a preset algorithm, a position and a posture of the first photographing device when the first image is captured, and a position and a posture of the second photographing device when the second image is photographed;

The processor is configured to generate a terrain surface based on a position and a posture of the first image and the first photographing device when the first image is captured;

The processor is configured to: perform projection and splicing processing on the second image on the terrain surface to obtain an output image based on a position and a posture of the second photographing device when the second image is captured.
The aircraft controller according to claim 130, wherein said first photographing device and said second photographing device are identical in a horizontal direction when said aircraft is flying at a fixed relative height with respect to a surface Shooting at the shooting interval.
The aircraft controller according to claim 130, wherein a photographing interval of said first photographing device and said second photographing device changes when said aircraft changes in height with respect to a surface.
The aircraft controller according to claim 132, wherein said first photographing device and said second photographing device are time-variant photographing intervals in a horizontal direction when said aircraft is flying at a uniform absolute height Shooting is performed wherein the shooting interval is associated with a pre-configured image overlay rate and the relative height of the aircraft to the surface.
A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the output image generating method of any one of claims 1-46.
A drone, characterized in that it comprises:

body;

a power system mounted to the fuselage for providing flight power;

a first photographing device and a second photographing device are mounted on the body for capturing an image, wherein an FOV of the first photographing device is greater than or equal to a preset threshold;

And an aircraft controller according to any of claims 93-133.