WO2019100214A1

WO2019100214A1 - Method, device, and unmanned aerial vehicle for generating output image

Info

Publication number: WO2019100214A1
Application number: PCT/CN2017/112189
Authority: WO
Inventors: 马岳文; 张明磊; 马东东; 赵开勇
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2017-11-21
Filing date: 2017-11-21
Publication date: 2019-05-31
Also published as: CN110073403A

Abstract

The embodiments of the present invention provide a method, device, and unmanned aerial vehicle for generating an output image. The method comprises: obtaining an image by an image capture device installed on an aircraft; on the basis of a preset image processing algorithm, calculating a position and attitude of the image capture device when the image was captured; and on the basis of the position and attitude, performing projection processing and image stitching processing on the image to obtain an output image. The method, device, and unmanned aerial vehicle provided by the embodiments of the invention can reduce equipment costs while producing a better stitched output image.

Description

Output image generation method, device and drone

Technical field

The present application relates to the field of UAV application technologies, and in particular, to an output image generation method, device, and drone.

Background technique

Digital Orthophoto Map (DOM) is a digital aerial image/remote sensing image (monochrome/color) that is scanned and processed by digital elevation model. The projection difference is corrected by pixel, and then image mosaic. The image generated by stitching according to the range of the frame. Since the image uses a real terrain surface as a mosaic projection surface, it has real geographic coordinate information, and the true distance can be measured on the image.

The method for generating digital orthophotos in the prior art mainly collects the position and posture of the shooting device when shooting images by using a global positioning system (GPS) and an inertial measurement unit (IMU) mounted on the shooting device, and according to the position and posture, The image is projected onto the estimated average elevation surface, and the digital orthophoto is obtained after splicing.

However, due to the high price of high-precision GPS and IMU, if high-precision GPS and IMU are used, the cost is high, and with lower precision GPS and IMU, although the cost of the equipment is reduced, it is based on low precision. The stitching effect of the image obtained by the position and posture is poor.

Summary of the invention

The embodiment of the invention provides an output image generation method, a device and a drone to obtain an output image with better stitching effect and reduce equipment cost.

A first aspect of the present invention provides a method for generating an output image, including:

Obtaining images obtained by shooting equipment carried on the aircraft;

Calculating a position and a posture of the photographing device when the image is captured based on a preset image processing algorithm;

Based on the position and the posture, the image is subjected to projection processing and image stitching processing to obtain an output image.

A second aspect of the embodiments of the present invention provides a ground station, including:

a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire an image captured by a photographing device mounted on the aircraft;

The processor is configured to: obtain, according to a preset image processing algorithm, a position and a posture of the photographing device when the image is captured;

The processor is further configured to perform a projection process and an image stitching process on the image to obtain an output image based on the position and the posture.

A third aspect of the embodiments of the present invention provides a controller, including:

A fourth aspect of an embodiment of the present invention provides a computer readable storage medium comprising instructions, when executed on a computer, causing a computer to execute the output image generating method of the first aspect described above.

A fifth aspect of the embodiments of the present invention provides a drone, including:

body;

a power system mounted to the fuselage for providing flight power;

a photographing device mounted on the body for capturing an image;

And the control device of the above third aspect.

The output image generating method, the device and the drone provided by the embodiment of the present invention obtain the image obtained by the shooting device mounted on the aircraft, and calculate the position of the shooting device when the image is captured based on the preset image processing algorithm. The posture is based on the position and posture of the photographing device when the image is captured, and the image is subjected to image processing and image stitching processing to obtain an output image. Since the position and posture of the photographing device when photographing the image are obtained by the preset image processing algorithm in the embodiment of the present invention, it is not necessary to mount the high-precision GPS and IMU on the aircraft to obtain a more accurate position and posture. , so that it can be reduced while obtaining a better spliced output image Equipment cost.

DRAWINGS

1 is a flowchart of an output image generating method provided by the present invention;

2 is a schematic diagram of a connection between a ground station and an aircraft according to an embodiment of the present invention;

FIG. 3 is a flowchart of an image projection method according to an embodiment of the present invention;

4a and 4b are schematic diagrams showing output images of two identical scenes provided by the present invention;

5a and FIG. 5b are schematic diagrams of output images in two identical scenarios according to an embodiment of the present invention;

FIG. 6 is a flowchart of a method for generating an output image according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a ground station according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a controller according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly described with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

It should be noted that when a component is referred to as being "fixed" to another component, it can be directly on the other component or the component can be present. When a component is considered to "connect" another component, it can be directly connected to another component or possibly a central component.

All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. The terminology used in the description of the present invention is for the purpose of describing particular embodiments and is not intended to limit the invention. The term "and/or" used herein includes any and all combinations of one or more of the associated listed items.

Some embodiments of the present invention are described in detail below with reference to the accompanying drawings. The features of the embodiments and examples described below can be combined with each other without conflict.

Embodiments of the present invention provide an output image generation method, which may be performed by a ground station Or a controller mounted on the drone to execute. The following embodiment is a detailed description of the ground station. The implementation manner of the controller is similar to that of the ground station, and is not described in this embodiment. Referring to FIG. 1 , FIG. 1 is a flowchart of a method for generating an output image according to the present invention. As shown in FIG. 1 , the method in this embodiment includes:

Step 101: Acquire an image captured by a shooting device mounted on the aircraft.

The ground station in this embodiment is a device having a computing function and/or processing capability, and the device may specifically be a remote controller, a smart phone, a tablet computer, a laptop computer, a watch, a wristband, and the like, and combinations thereof.

The aircraft in this embodiment may specifically be a drone equipped with a photographing device, a helicopter, a manned fixed-wing aircraft, a hot air balloon, or the like.

As shown in FIG. 2, the ground station 21 and the aircraft 22 can be connected through an Application Programming Interface (API) 23, but are not limited to being connected through an API. Specifically, the ground station 21 and the aircraft 22 can be connected by wire or wirelessly, for example, by at least one of the following methods: WIreless-Fidelity (WI-FI), Bluetooth, software defined radio (software defined radio) , referred to as SDR) or other custom protocols.

Optionally, in this embodiment, the aircraft can perform automatic cruising and photographing according to a predetermined route, and can also perform cruising and photographing under the control of the ground station.

In this embodiment, the shooting device performs shooting according to a preset shooting interval or distance interval, and images captured by adjacent shooting moments have overlapping portions, wherein the size of the overlapping portion can be set according to needs, for example, by setting The corresponding shooting interval or distance interval is used to obtain the size of the overlap portion required, but it is not limited to determining the size of the image overlapping portion by setting the shooting interval or the distance interval.

For example, the photographing device of the aircraft in this embodiment can be photographed in the following possible ways:

In one possible way: when the aircraft is flying at a fixed relative height relative to the surface, the aircraft's camera is photographed at the same shooting interval in the horizontal direction.

In another possible manner: when the altitude of the aircraft changes relative to the surface, the shooting interval of the aircraft's shooting device changes, for example, when the aircraft is flying at a uniform absolute altitude, the aircraft's shooting device changes in a horizontal direction. Shooting interval, where, specifically, The shooting interval can be determined based on the pre-configured image overlap rate and the relative height of the aircraft and the surface. Of course, this is merely an illustration and not a limitation of the invention. Optionally, in the embodiment, the ground station can obtain the image obtained by the shooting device by using the following possible ways:

In one possible way, the aircraft transmits the image captured by the photographing device to the ground station in real time through the API between it and the ground station.

In another possible manner, the aircraft transmits the image captured by the photographing device within the preset time interval to the ground station at preset time intervals.

In yet another possible manner, after the cruise is over, the aircraft transmits the images captured by the photographing device during the entire cruise to the ground station.

Specifically, based on the above manner, the aircraft may transmit the image captured by the photographing device to the ground station in the form of code stream data, or may send the image to the ground station in the form of a thumbnail, but according to the computing power of the aircraft and the ground station, The resolution of the returned stream data or thumbnail is not specifically limited and may be the original image. In this embodiment, taking the form of a thumbnail as an example, when the image is sent to the ground station in the form of a thumbnail, the ground station can display the received thumbnail so that the user can clearly see the image obtained by the real-time shooting. .

Step 102: Calculate, according to a preset image processing algorithm, a position and a posture of the photographing device when the image is captured.

The preset image processing method in this embodiment may specifically be a motion recovery structure algorithm, an aerial triangulation algorithm, and a simultaneous localization and mapping (SLAM) algorithm. In this embodiment, the SLAM algorithm is taken as an example to calculate the position and posture of the photographing device when the image is captured. The method for calculating the position and posture of the photographing device when the image is taken by the SLAM algorithm is similar to the prior art, and will not be described herein.

It should be noted here that in the existing algorithm, the SLAM algorithm calculates the position and posture of the photographing device based on the matching of image feature points, and therefore, the position and posture obtained by the SLAM calculation are relative positions and relatives in the shooting scene. attitude.

Optionally, in order to make the position and posture obtained by the above calculation correspond to the world coordinate system, the position and posture corresponding to the image are more practical reference values. In this embodiment, the aircraft sends the image to the ground station. The GPS information of the shooting position of the image is transmitted to the ground station. The ground station converts the calculated position and posture into the world based on the GPS information corresponding to the image. Position and attitude in coordinates; in another embodiment, world coordinates may be acquired using a method of identifying the identified identifier, and the calculated position and pose are converted to positions and poses in world coordinates.

Step 103: Perform projection processing and image splicing processing on the image based on the position and the posture to obtain an output image.

Optionally, the output image involved in this embodiment may be specifically an orthophoto, such as an orthophoto map or other image with real geographic coordinate information obtained according to orthographic projection.

Specifically, the image projection method based on the position and posture of the photographing device (which may be the relative position and the relative posture calculated by the SLAM algorithm, or the position and posture in the world coordinate system) in the embodiment includes the following:

In one possible implementation, the image is projected onto the average elevation surface according to the position and attitude of the photographing device by estimating the average elevation surface. The way to obtain the estimated average elevation surface is similar to the prior art and will not be described here.

In another possible implementation, the terrain surface is fitted by a point cloud fitting method, and the image is projected onto the terrain surface according to the position and posture of the photographing device. Specifically, FIG. 3 is a flowchart of a method for image projection according to an embodiment of the present invention. As shown in FIG. 3 , in this implementation manner, a method for projecting an image includes:

Step 301: Calculate a semi-dense or dense point cloud of the image based on the position and the posture, or calculate a sparse point cloud of the image based on a SLAM algorithm.

Step 302: Fit the terrain cloud based on the calculated point cloud.

Step 303: Project the image onto the terrain surface based on the position and posture of the image.

The method for calculating the image dense point cloud, the semi-dense point cloud, or the sparse point cloud in the embodiment of FIG. 3 may be any method in the prior art, which is not specifically limited in this embodiment.

In the actual scene, after obtaining the position and posture of the photographing device based on the image calculation, the image may be first projected onto the surface of the fitted terrain based on the method shown in FIG. 3, and then the image on the surface of the terrain is stitched to obtain a relatively rough image. Output image.

Optionally, when performing image projection processing, the point cloud obtained by the above calculation may be first divided according to ground points and non-ground points, and the terrain surface is fitted according to ground points in the point cloud. Step, and then project the image onto the terrain surface according to the position and posture of the shooting device when shooting the image.

Further, if it is desired to obtain a more accurate and clear output image, after step 301, the point cloud obtained by the above calculation or/and the position and posture obtained by the above calculation are optimized to obtain a point cloud meeting the preset quality condition. Or / and position and attitude that meet the preset accuracy conditions. Then, the optimized point cloud is divided into ground points and non-ground points, so that the digital elevation model is generated by the ground point fitting in the optimized point cloud, and the digital elevation model is projected as the projected terrain surface. Certainly, the timing of dividing the point cloud into the ground point and the non-ground point in this embodiment is not uniquely limited. In fact, the point cloud may be first divided, and then the point cloud or/and the position and posture are optimized. This embodiment does not specifically limit it.

Optionally, the splicing processing method of the image in this embodiment may be one of the following methods: a direct overlay method, a panoramic image splicing method, a method for selecting a region closest to the image center in each region of the final image, and a cost based method. The splicing method of the function. In this embodiment, a method of fitting a terrain surface by using a point cloud is taken as an example to determine a projection surface, and a projection on the projection surface is spliced based on a cost function splicing method, that is, a distance from the projected pixel to the photographing device is used as a constraint construction cost. The function, based on the cost function, splicing the projection of the image onto the surface of the terrain, so that the color difference on both sides of the splicing line is minimized.

Optionally, in this embodiment, the ground station can process the received image by using the following two working modes:

In one possible approach, the ground station processes the received image in a ready-to-go process. That is to say, when the ground station is in the cruising of the aircraft, the received image is processed to obtain a semi-dense point cloud, a dense point cloud or a sparse point cloud of the image. In this way, the ground station updates the semi-dense point cloud, dense point cloud or sparse point cloud obtained by the processing for each image received. In addition, it should be noted that the above-mentioned processing method is not only the processing method included in the literal meaning, but depends on the processing speed of the ground station. If the processing speed of the ground station can support the reception or processing, then The ground station processes the image immediately after receiving the image. If the processing speed of the ground station is insufficient to support the instant processing of the image, the ground station sequentially processes the received image. Specifically, the ground station can follow the image. The receiving sequence is processed, and may be processed according to the storage order of the images, and may be processed according to other custom processing sequences, which is not specifically limited in this embodiment.

In another possible processing mode, when the ground station is in the cruising of the aircraft, only the image captured by the shooting device is received, and when the aircraft ends the cruise shooting, the received image is processed centrally.

Optionally, when splicing the image projection, the global color adjustment and/or brightness adjustment of the calculated point cloud may be first performed to achieve the purpose of significantly improving image quality, and further, based on the adjusted Projecting the image, constructing the cost function by using the distance from the projected pixel to the photographing device as a constraint, and stitching the projection of the image onto the surface of the terrain based on the cost function, so that the color difference on both sides of the stitching line is minimized, so that the integrity can be obtained. Better output image.

Further, in order to avoid the influence of the non-ground point splicing in the point cloud (the non-ground point may cause the splicing misalignment), in the embodiment, when constructing the cost function, the non-ground point in the point cloud may also be excluded, so that the splicing is performed. The line can automatically avoid non-ground areas, resulting in a better visual output image.

Specifically, FIG. 4a and FIG. 4b are schematic diagrams of output images of two identical scenes provided by the present invention, wherein FIG. 4a is an output image obtained by using an estimated elevation surface as a projection surface, and FIG. 4b is formed by a point cloud fitting. The output image obtained by the terrain surface as the projection surface is spliced by the cost function method. As shown in Fig. 4a, in Fig. 4a, since the estimated elevation surface is used as the projection surface, since the elevation surface cannot accurately fit the terrain surface, the output image of Fig. 4a produces a severe stitching misalignment. In Figure 4b, because the terrain surface formed by point cloud fitting is used as the projection surface, the terrain surface can be fitted more accurately, and the cost function can be used to minimize the chromatic aberration on both sides of the splicing line, so the output is obtained. There is no obvious dislocation phenomenon in the image, and the overall output image is better overall. Therefore, in the embodiment of the present invention, the terrain surface is fitted by the point cloud, and the cost function is used to perform the splicing processing, which can solve the problem of the output image mosaic misalignment.

Optionally, in order to make the entire output image have a better visual effect, after the image is projected onto the terrain surface, the color and brightness of the projection on the terrain surface may be adjusted based on a preset strategy. This enables a better stitching effect in the subsequent stitching process.

For example, FIG. 5a and FIG. 5b are schematic diagrams of output images in two identical scenes according to an embodiment of the present invention. The projection on the terrain surface in FIG. 5a is not processed by color and brightness, and therefore, the entire output in FIG. 5a The integrity of the image in terms of color and brightness is not very good, depending on The effect is poor, and in Figure 5b, the brightness and color of the projection on the terrain surface are processed before the splicing, so the resulting output image is better in color and brightness, and the visual effect is better. it is good. Therefore, the embodiment of the present invention can effectively improve the visual effect of the output image by performing color and brightness processing on the projection on the terrain surface before the splicing process.

Optionally, in this embodiment, the step of displaying an output image may be further included, wherein the output image may be an orthophoto. Because orthophotos are measurable, they can provide a large amount of geographic information, especially in the context of natural disasters such as earthquakes, as well as in agriculture, mapping, and transportation planning.

The output image generating method provided by the embodiment obtains an image captured by a photographing device mounted on the aircraft, and calculates a position and a posture of the photographing device when the image is captured based on a preset image processing algorithm, so that the photographing device is based on the photographing device. The position and posture of the image are taken, and the image is processed by image processing and image stitching to obtain an output image. Since the position and posture of the photographing device when photographing the image are obtained by the preset image processing algorithm in the embodiment of the present invention, it is not necessary to mount the high-precision GPS and IMU on the aircraft to obtain a more accurate position and posture. Therefore, it is possible to reduce the equipment cost while obtaining a better spliced output image.

In addition, since the present embodiment can generate real-time orthophotos, the integrated solution from image acquisition to orthophoto generation is greatly improved in comparison with the existing non-real-time orthophoto generation solution. Job productivity, non-real-time orthophoto generation solutions typically require images to be imported from the aircraft to the computer and manipulated by the software for processing, and typically require several hours of processing time. The orthophoto generation scheme provided in this embodiment can acquire a higher-precision orthophoto of the survey area after the aircraft data acquisition operation ends.

The embodiment of the present invention provides an output image generating method. In this embodiment, the photographing device may be specifically a camera. As shown in FIG. 6 , in the method, an operator sets a cruise area and a cruise route for the aircraft through the ground station, and the aircraft collects images according to the flight route in the cruise area, and the aircraft collects the obtained images to take a thumbnail or The code stream is sent to the ground station. After receiving the image, the ground station initializes the SLAM algorithm and generates the initial semi-dense point of the shooting scene. Further, through the SLAM algorithm, the position and posture of the camera when the image is captured are calculated and generated by dense matching. The semi-dense point of the image. In the update shooting After the semi-dense point of the scene, the ground station fits the terrain surface based on the semi-dense points and projects the received image onto the fitted terrain surface according to the position and attitude at the time of shooting. Then the cost function is constructed based on the principle of minimum color difference on both sides of the stitching line. The cost function is used to find the optimal stitching line to splicing the image on the terrain surface. Further, the ground station may further determine whether the aircraft has completed image acquisition according to the time of the aircraft cruise or the number of images returned by the aircraft, and if so, the camera position and posture corresponding to the acquired image based on the SLAM algorithm, and The semi-dense point cloud is optimized to obtain a position and attitude that meets the preset accuracy requirements, as well as a point cloud that meets the preset quality requirements. Further, the ground station classifies the optimized point cloud, divides the point cloud into ground points and non-ground points, and re-fitting the terrain surface based on the divided ground points to re-project the image onto the re-fitted terrain surface. on. After that, in order to obtain better visual effects, the ground station can also perform global color adjustment on the projection on the surface of the terrain to ensure color consistency, and further construct a cost function to automatically wrap around when selecting the stitching line. Passing through non-ground points (such as buildings and other objects), the resulting output image will not have the problem of misalignment, and has a good visual effect.

The specific implementation manner and beneficial effects of the method provided in this embodiment are similar to those in the embodiment of FIG. 1, and are not described herein again.

The embodiment of the invention provides a ground station, which may be the ground station described in the above embodiment. FIG. 7 is a schematic structural diagram of a ground station according to an embodiment of the present invention. As shown in FIG. 7, the ground station 10 includes: a communication interface 11, one or more processors 12; and one or more processors work independently or in cooperation. The interface 11 is connected to the processor 12; the communication interface 11 is configured to: acquire an image captured by a photographing device mounted on the aircraft; and the processor 12 is configured to: obtain, according to a preset image processing algorithm, the photographing device to capture the image The position and posture of the time; the processor 12 is further configured to: perform projection processing and image stitching processing on the image to obtain an output image based on the position and the posture.

Optionally, the communication interface 11 is configured to: acquire code stream data of an image captured by a photographing device mounted on an aircraft.

Optionally, the communication interface 11 is configured to: acquire a thumbnail of an image captured by a photographing device mounted on the aircraft.

Optionally, the ground station further includes a display component 13, the display component 13 and the location The processor 12 is communicatively coupled; the display component 13 is configured to: display the acquired thumbnail.

Optionally, the communication interface 11 is further configured to: acquire GPS information of the photographing device when the image is captured; the processor 12 is further configured to: based on the GPS information corresponding to the image, The position is converted to a position in the world coordinate system, and the posture is converted into a posture in the world coordinate system.

Optionally, the processor 12 is configured to: calculate a position and a posture of the photographing device when the image is captured, based on an instant positioning and map construction SLAM algorithm.

Optionally, the processor 12 is configured to: construct a cost function, and perform splicing processing on the projection of the image onto the surface of the terrain based on the cost function.

Optionally, the processor 12 is further configured to: perform optimization processing on the point cloud to obtain a point cloud that meets a preset quality condition; and the processor 12 is configured to: form a terrain based on the optimized point cloud fitting surface.

Optionally, the processor 12 is configured to: extract a ground point from the optimized point cloud; and fit the terrain surface based on the ground point.

Optionally, the processor 12 is configured to: perform optimization processing on the position and the posture, and obtain a position and a posture that meet a preset accuracy condition.

Optionally, the processor 12 is configured to: perform color and/or brightness adjustment on a projection of the image on the terrain surface based on a preset policy.

Optionally, the output image includes an orthophoto.

Optionally, the display component 13 is configured to display the orthophoto.

Optionally, when the aircraft is flying at a fixed relative height with respect to the ground surface, the processor 12 is configured to: control the shooting device of the aircraft to shoot at the same shooting interval in the horizontal direction.

Optionally, when the aircraft is changed in height relative to the surface, the processor 12 is configured to: control the shooting device of the aircraft to change the shooting interval for shooting.

Optionally, when the aircraft is flying at a uniform absolute height, the processor 12 is configured to: the photographing device that controls the aircraft is photographed in a horizontal direction at a time-lapse photographing interval, wherein the photographing interval is The overlap rate with the pre-configured image and the relative height of the aircraft to the surface.

The ground station provided in this embodiment can perform the technical solution of the embodiment of FIG. 1 and its execution manner Similar to the beneficial effects, it will not be described here.

The embodiment of the present invention further provides a ground station. The ground station is based on the embodiment of FIG. 7. The processor 12 is configured to: calculate a semi-dense or dense point cloud of the image based on the position and the posture. Or calculating a sparse point cloud of the image based on the SLAM algorithm; fitting the terrain cloud based on the calculated point cloud; and projecting the image onto the terrain surface based on the position and orientation of the image.

The ground station provided by this embodiment can perform the technical solution of the embodiment of FIG. 3, and the execution manner and the beneficial effects are similar, and details are not described herein again.

The embodiment of the invention provides a controller. Referring to FIG. 8, FIG. 8 is a schematic structural diagram of a controller according to an embodiment of the present invention. As shown in FIG. 8, the controller 20 includes: a communication interface 21, one or more processors 22; and one or more processors alone or Working in cooperation, the communication interface 21 is connected to the processor 22; the communication interface 21 is configured to: acquire an image captured by a photographing device mounted on the aircraft; and the processor 22 is configured to: calculate, according to a preset image processing algorithm, the photographing device The position and posture of the image is taken; the processor 22 is further configured to: perform projection processing and image stitching processing on the image based on the position and the posture to obtain an output image.

Optionally, the communication interface 21 is configured to: acquire code stream data of an image captured by a photographing device mounted on the aircraft.

Optionally, the communication interface 21 is configured to: acquire a thumbnail of an image captured by a photographing device mounted on the aircraft.

Optionally, the communication interface 21 is further configured to: acquire GPS information when the imaging device captures the image; and the processor 22 is further configured to: based on the GPS information corresponding to the image, The position is converted to a position in the world coordinate system, and the posture is converted into a posture in the world coordinate system.

Optionally, the processor 22 is configured to: calculate a position and a posture of the photographing device when the image is captured based on a real-time positioning and map construction SLAM algorithm.

Optionally, the processor 22 is configured to: construct a cost function, and perform splicing processing on the projection of the image onto the surface of the terrain based on the cost function.

Optionally, the processor 22 is further configured to: perform optimization processing on the point cloud to obtain a character a point cloud in combination with a preset quality condition; the processor 22 is configured to form a terrain surface based on the optimized point cloud fitting.

Optionally, the processor 22 is configured to: extract a ground point from the optimized point cloud; and fit the terrain surface based on the ground point.

Optionally, the processor 22 is configured to: perform optimization processing on the position and the posture, and obtain a position and a posture that meet a preset accuracy condition.

Optionally, the processor 22 is configured to perform color and/or brightness adjustment on a projection of the image on the terrain surface based on a preset policy.

Optionally, the output image includes an orthophoto.

Optionally, when the aircraft is flying at a fixed relative height with respect to the ground surface, the processor 22 is configured to: control the shooting device of the aircraft to shoot at the same shooting interval in the horizontal direction.

Optionally, when the aircraft is changed in height relative to the surface, the processor 22 is configured to: control the shooting device of the aircraft to change the shooting interval for shooting.

Optionally, when the aircraft is flying at a uniform absolute height, the processor 22 is configured to: the photographing device that controls the aircraft is photographed in a horizontal direction at a time-lapse photographing interval, wherein the photographing interval is The overlap rate with the pre-configured image and the relative height of the aircraft to the surface.

The controller provided in this embodiment can perform the technical solution of the embodiment of FIG. 1 , and the execution manner and the beneficial effects are similar, and details are not described herein again.

The embodiment of the present invention further provides a controller, based on the embodiment of FIG. 8, the processor 22 is configured to: calculate a semi-dense or dense point cloud of the image based on the position and the posture. Or calculating a sparse point cloud of the image based on the SLAM algorithm; fitting the terrain cloud based on the calculated point cloud; and projecting the image onto the terrain surface based on the position and orientation of the image.

The controller provided in this embodiment can perform the technical solution of the embodiment of FIG. 3, and the execution manner and the beneficial effects are similar, and details are not described herein again.

Embodiments of the present invention provide a computer readable storage medium including instructions when it is in a calculation When the machine is running, the computer is caused to execute the output image generating method provided by the above embodiment.

Embodiments of the present invention provide a drone. The drone includes a fuselage; a power system mounted to the body for providing flight power; a photographing device mounted to the body for capturing an image; and a controller as described in the above embodiments.

The execution mode and the beneficial effects of the unmanned aerial vehicle provided in this embodiment are the same as those of the controller in the foregoing embodiment, and are not described herein again.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above-described integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The above software functional unit is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform the methods of the various embodiments of the present invention. Part of the steps. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes. .

Those skilled in the art can clearly understand that for the convenience and simplicity of the description, only the above The division of each functional module is illustrated. In practical applications, the above function assignment can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. . For the specific working process of the device described above, refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that The technical solutions described in the foregoing embodiments may be modified, or some or all of the technical features may be equivalently replaced; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present invention. range.

Claims

An output image generating method, comprising:

Obtaining images obtained by shooting equipment carried on the aircraft;

Calculating a position and a posture of the photographing device when the image is captured based on a preset image processing algorithm;

Based on the position and the posture, the image is subjected to projection processing and image stitching processing to obtain an output image.
The method according to claim 1, wherein the acquiring an image obtained by the photographing device mounted on the aircraft comprises:

Obtain the code stream data of the image captured by the shooting device mounted on the aircraft.
The method according to claim 1, wherein the acquiring an image obtained by the photographing device mounted on the aircraft comprises:

Obtain a thumbnail of the image captured by the shooting device mounted on the aircraft.
The method of claim 3, wherein the method further comprises:

The obtained thumbnail is displayed.
The method of claim 1 further comprising:

Obtaining GPS information of the photographing device when the image is captured;

The method further includes: calculating, according to a preset image processing algorithm, a position and a posture of the photographing device when the image is captured, the method further comprising:

Based on the GPS information corresponding to the image, the position is converted into a position in a world coordinate system, and the posture is converted into a posture in a world coordinate system.
The method according to claim 1, wherein the calculating the position and posture of the photographing device when the image is captured based on a preset image processing algorithm comprises:

Based on the real-time positioning and map construction SLAM algorithm, the position and posture of the photographing device when the image is captured are calculated.
The method according to any one of claims 1 to 6, wherein the projecting the image based on the position and the posture comprises:

Obtaining a semi-dense or dense point cloud of the image based on the position and the posture, or calculating a sparse point cloud of the image based on a SLAM algorithm;

A point cloud fitted to the terrain surface based on the calculation;

The image is projected onto the terrain surface based on the position and orientation of the image.
The method according to claim 7, wherein the image splicing processing on the image based on the position and the posture comprises:

A cost function is constructed to stitch the projections of the image onto the surface of the terrain based on the cost function.
The method according to claim 7, wherein the calculating a semi-dense or dense point cloud of the image based on the position and the posture, or calculating a sparse point of the image based on a SLAM algorithm After the cloud, the method further includes:

Optimizing the point cloud to obtain a point cloud that meets preset quality conditions;

The point cloud based on the calculation is fitted to the terrain surface, including:

The terrain surface is formed based on the optimized point cloud fitting.
The method according to claim 9, wherein the forming a terrain surface based on the optimized point cloud fitting comprises:

Extract ground points from the optimized point cloud;

The terrain surface is fitted based on the ground point.
The method according to claim 7, wherein the calculating a semi-dense or dense point cloud of the image based on the position and the posture, or calculating a sparse point of the image based on a SLAM algorithm After the cloud, the method further includes:

The position and the posture are optimized to obtain a position and a posture that meet the preset accuracy condition.
The method according to claim 7, wherein after the image is projected onto the terrain surface based on the position and orientation of the image, the method further comprises:

The color and/or brightness adjustment of the projection of the image on the terrain surface is performed based on a preset strategy.
The method of any of claims 1-12, wherein the output image comprises an orthophoto.
The method of claim 13 wherein the method further comprises:

The orthophoto is displayed.
The method according to claim 1, wherein said photographing apparatus photographs in the same direction in the horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface. Shoot at intervals.
The method of claim 1 wherein the photographing interval of the photographing device changes when the aircraft is changed in height relative to the surface.
The method according to claim 16, wherein when the aircraft is flying at a uniform absolute height, the photographing apparatus performs photographing at a time-varying photographing interval in a horizontal direction, wherein the photographing interval is in advance The configured image overlap rate and the relative height of the aircraft to the surface.
A ground station, comprising: a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire an image captured by a photographing device mounted on the aircraft;

The processor is configured to: obtain, according to a preset image processing algorithm, a position and a posture of the photographing device when the image is captured;

The processor is further configured to perform a projection process and an image stitching process on the image to obtain an output image based on the position and the posture.
The ground station according to claim 18, wherein the communication interface is configured to: acquire code stream data of an image captured by a photographing device mounted on the aircraft.
The ground station according to claim 18, wherein the communication interface is configured to: acquire a thumbnail of an image captured by a photographing device mounted on the aircraft.
The ground station of claim 20, wherein said ground station further comprises a display component, said display component being communicatively coupled to said processor;

The display component is configured to: display the acquired thumbnail.
The ground station according to claim 18, wherein the communication interface is further configured to: acquire GPS information of the photographing device when the image is captured;

The processor is further configured to: convert the position into a position in a world coordinate system based on the GPS information corresponding to the image, and convert the posture into a posture in a world coordinate system.
The ground station according to claim 18, wherein the processor is configured to: calculate a position and a posture of the photographing device when the image is captured based on an instant positioning and map construction SLAM algorithm.
A ground station according to any one of claims 18 to 23, wherein said said The processor is used to:

Obtaining a semi-dense or dense point cloud of the image based on the position and the posture, or calculating a sparse point cloud of the image based on a SLAM algorithm;

A point cloud fitted to the terrain surface based on the calculation;

The image is projected onto the terrain surface based on the position and orientation of the image.
The ground station according to claim 24, wherein said processor is configured to: construct a cost function, and splicing said projection of said image onto said surface of said terrain based on said cost function.
The ground station of claim 24, wherein the processor is further configured to:

Optimizing the point cloud to obtain a point cloud that meets preset quality conditions;

The processor is configured to form a terrain surface based on the optimized point cloud fitting.
The ground station of claim 26 wherein said processor is operative to:

Extract ground points from the optimized point cloud;

The terrain surface is fitted based on the ground point.
The ground station of claim 24 wherein said processor is operative to:

The position and the posture are optimized to obtain a position and a posture that meet the preset accuracy condition.
The ground station according to claim 24, wherein the processor is configured to perform color and/or brightness adjustment on a projection of the image on the terrain surface based on a preset policy.
A ground station according to any of claims 18-29, wherein the output image comprises an orthophoto.
The ground station of claim 30 wherein the display component is for displaying the orthophoto.
The ground station according to claim 18, wherein said processor is configured to: control a photographing device of said aircraft to shoot in the same direction in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface Shoot at intervals.
The ground station according to claim 18, wherein said processor is configured to: control a change of a photographing device of said aircraft when said aircraft is changed in height relative to a surface Shooting at the shooting interval.
The ground station according to claim 33, wherein when the aircraft is flying at a uniform absolute altitude, the processor is configured to: control a photographing device of the aircraft to perform a time-lapse photographing interval in a horizontal direction Shooting, wherein the shooting interval is associated with a pre-configured image overlap rate and a relative height of the aircraft to the surface.
A controller for an aircraft, characterized by a communication interface, one or more processors; the one or more processors operating separately or in cooperation, the communication interface being coupled to the processor;

The communication interface is configured to: acquire an image captured by a photographing device mounted on the aircraft;

The processor is configured to: obtain, according to a preset image processing algorithm, a position and a posture of the photographing device when the image is captured;

The processor is further configured to perform a projection process and an image stitching process on the image to obtain an output image based on the position and the posture.
The controller according to claim 35, wherein the communication interface is configured to: acquire code stream data of an image captured by a photographing device mounted on the aircraft.
The controller according to claim 35, wherein the communication interface is configured to: acquire a thumbnail of an image captured by a photographing device mounted on the aircraft.
The controller according to claim 35, wherein the communication interface is further configured to: acquire GPS information of the photographing device when the image is captured;

The processor is further configured to: convert the position into a position in a world coordinate system based on the GPS information corresponding to the image, and convert the posture into a posture in a world coordinate system.
The controller according to claim 35, wherein the processor is configured to: calculate a position and a posture of the photographing device when the image is captured based on an instant positioning and map construction SLAM algorithm.
The controller according to any one of claims 35 to 39, wherein the processor is configured to:

Obtaining a semi-dense or dense point cloud of the image based on the position and the posture, or calculating a sparse point cloud of the image based on a SLAM algorithm;

A point cloud fitted to the terrain surface based on the calculation;

The image is projected onto the terrain surface based on the position and orientation of the image.
The controller according to claim 40, wherein said processor is configured to: construct a cost function, and perform splicing processing on said projection of said image onto said surface of said terrain based on said cost function.
The controller of claim 40, wherein the processor is further configured to:

Optimizing the point cloud to obtain a point cloud that meets preset quality conditions;

The processor is configured to form a terrain surface based on the optimized point cloud fitting.
The controller of claim 42 wherein said processor is operative to:

Extract ground points from the optimized point cloud;

The terrain surface is fitted based on the ground point.
The controller of claim 40 wherein said processor is operative to:

The position and the posture are optimized to obtain a position and a posture that meet the preset accuracy condition.
The controller according to claim 40, wherein the processor is configured to perform color and/or brightness adjustment on a projection of the image on the terrain surface based on a preset policy.
The controller of any of claims 35-45, wherein the output image comprises an orthophoto.
The controller according to claim 35, wherein said processor is configured to: control said photographing apparatus to perform at the same photographing interval in a horizontal direction when said aircraft is flying at a fixed relative height with respect to the earth's surface. Shooting.
The controller according to claim 35, wherein said processor is adapted to: control said photographing device to change a photographing interval for photographing when said aircraft is changed in height relative to a surface.
The controller according to claim 48, wherein when the aircraft is flying at a uniform absolute height, the processor is configured to: control the photographing device to shoot in a horizontal direction at a time-lapse shooting interval, Wherein, the shooting interval is associated with a pre-configured image overlap rate and a relative height of the aircraft and the surface.
A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the output image generation method of any of claims 1-17.
A drone, characterized in that it comprises:

body;

a power system mounted to the fuselage for providing flight power;

a photographing device mounted on the body for capturing an image;

And a controller according to any of claims 35-49.