WO2020103110A1

WO2020103110A1 - Image boundary acquisition method and device based on point cloud map and aircraft

Info

Publication number: WO2020103110A1
Application number: PCT/CN2018/117038
Authority: WO
Inventors: 王涛; 马东东; 张明磊; 刘政哲; 李鑫超; 闫光; 杨志华
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2018-11-22
Filing date: 2018-11-22
Publication date: 2020-05-28
Also published as: CN110770791A

Abstract

Disclosed are an image boundary acquisition method and device based on point cloud map, an aircraft and a storage medium. The method comprises the following steps: obtaining a point cloud map containing semantics (S201); determining, according to the semantics on the point cloud map, respective image regions of different semantics on the point cloud map (S202). The described method can realize the automatic segmentation of image regions, satisfy the demands for automatic and intelligent classification of image regions, and improve the accuracy of image segmentation.

Description

Method, equipment and aircraft for acquiring image boundary based on point cloud map

Technical field

The invention relates to the technical field of control, and in particular to a method, device and aircraft for acquiring image boundaries based on a point cloud map.

Background technique

With the development of aircraft technology, currently aircraft (such as drones) have been widely used to perform various types of operational tasks (such as aerial photography, agricultural plant protection, surveys, etc.), of which, the most widely used aerial photography technology on aircraft . Taking an aircraft mounted with a shooting device as an example, the traditional aerial photography technology cannot automatically divide the image areas of different categories in the captured image during the shooting process, which affects the aircraft to perform operational tasks to a certain extent. Therefore, how to classify image regions more effectively has become the focus of research.

Summary of the invention

Embodiments of the present invention provide an image boundary acquisition method, device, and aircraft based on a point cloud map, which can automatically divide an image area to meet the needs of automation and intelligence for classifying image areas.

In a first aspect, an embodiment of the present invention provides a method for acquiring an image boundary based on a point cloud map. The method includes:

Get a point cloud map with semantics;

According to the semantics on the point cloud map, each image area with different semantics on the point cloud map is determined.

In a second aspect, an embodiment of the present invention provides a route planning method based on a point cloud map. The method includes:

Get a point cloud map with semantics;

According to the semantics on the point cloud map, determine each image area with different semantics on the point cloud map;

Plan flight routes according to the semantics of each image area on the point cloud map;

Controlling the aircraft to fly according to the flight path.

In a third aspect, an embodiment of the present invention provides an image boundary acquisition device based on a point cloud map, including a memory and a processor;

The memory is used to store program instructions;

The processor executes the program instructions stored in the memory. When the program instructions are executed, the processor is used to perform the following steps:

Get a point cloud map with semantics;

According to a fourth aspect, an embodiment of the present invention provides a route planning device based on a point cloud map, including a memory and a processor;

The memory is used to store program instructions;

Get a point cloud map with semantics;

Controlling the aircraft to fly according to the flight path.

According to a fifth aspect, an embodiment of the present invention provides an aircraft, including:

body;

A power system provided on the fuselage for providing flight power;

The processor is used to obtain a point cloud map containing semantics; according to the semantics on the point cloud map, determine each image area with different semantics on the point cloud map.

According to a sixth aspect, an embodiment of the present invention provides another aircraft, including:

body;

A power system provided on the fuselage for providing flight power;

A processor for acquiring a point cloud map containing semantics; determining each image area with different semantics on the point cloud map according to the semantics on the point cloud map; Plan a flight route; control the aircraft to fly according to the flight route.

According to a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium that stores a computer program, which when executed by a processor implements a point cloud-based map as described in the first aspect above Image boundary acquisition method or the route planning method based on point cloud map described in the second aspect.

In the embodiment of the present invention, an image boundary acquisition device based on a point cloud map can acquire a point cloud map containing semantics; according to the semantics on the point cloud map, each image area with different semantics on the point cloud map is determined. This method can automatically divide the image area to meet the needs of automation and intelligence to classify the image area.

BRIEF DESCRIPTION

In order to more clearly explain the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some of the present invention. For the embodiment, for those of ordinary skill in the art, without paying any creative labor, other drawings may be obtained based on these drawings.

1 is a schematic diagram of a working scene of an image boundary acquisition system based on a point cloud map provided by an embodiment of the present invention;

2 is a schematic flowchart of an image boundary acquisition method based on a point cloud map provided by an embodiment of the present invention;

Figure 3.1 is a schematic diagram of an etching operation provided by an embodiment of the present invention;

Figure 3.2 is a schematic diagram of an expansion operation provided by an embodiment of the present invention;

4 is a schematic flowchart of a route planning method based on a point cloud map provided by an embodiment of the present invention;

5 is a schematic diagram of an interface of a point cloud map provided by an embodiment of the present invention;

Figure 6.1 is a schematic diagram of an orthophoto image interface provided by an embodiment of the present invention;

FIG. 6.2 is a schematic diagram of another point cloud map interface provided by an embodiment of the present invention;

Figure 6.3 is a schematic diagram of an interface of a point cloud map for marking obstacles provided by an embodiment of the present invention;

7 is a schematic structural diagram of an image boundary acquisition device based on a point cloud map provided by an embodiment of the present invention;

8 is a schematic structural diagram of a route planning device based on a point cloud map provided by an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

The following describes some embodiments of the present invention in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and the features in the embodiments can be combined with each other.

The method for acquiring an image boundary based on a point cloud map provided by an embodiment of the present invention may be performed by an image boundary acquiring system based on a point cloud map, the image boundary acquiring system based on a point cloud map includes an image boundary acquiring based on a point cloud map For the device and the aircraft, a two-way communication connection can be established between the point cloud map-based image boundary acquisition device and the aircraft for two-way communication. In some embodiments, the point cloud map-based image boundary acquisition device may be set on an aircraft (such as a drone) equipped with a load (such as a camera, infrared detection device, surveying instrument, etc.). In other embodiments, the point cloud map-based image boundary acquisition device may also be provided on other movable devices, such as autonomous devices such as robots, unmanned vehicles, and unmanned boats. In some embodiments, the point cloud map-based image boundary acquisition device may be a component of an aircraft, that is, the aircraft includes the point cloud map-based image boundary acquisition device; in other embodiments, the based The point cloud map image boundary acquisition device can also be spatially independent of the aircraft. The following describes an example of an embodiment of a method for acquiring an image boundary based on a point cloud map for an aircraft with reference to the drawings.

In the embodiment of the present invention, an image boundary acquisition device based on a point cloud map may obtain a point cloud map containing semantics and determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map.

In one embodiment, when the image boundary acquisition device based on the point cloud map determines each image area with different semantics on the point cloud map according to the semantics on the point cloud map, the image boundary acquisition device may Determine the image areas with continuous and identical semantics on the point cloud map, and perform edge processing operations on the image areas with continuous and identical semantics to obtain image areas with different semantics on the point cloud map. In some embodiments, the edge processing operation includes a forward edge processing operation and / or a reverse edge processing operation.

In some embodiments, the forward edge processing operation and / or the reverse edge processing operation can eliminate noise, segment independent image elements, connect adjacent elements in the image, and find obvious maxima regions in the image Or the minimum area, find the gradient of the image to achieve the segmentation of the image. In some embodiments, the forward edge processing operation may be that the highlighted part in the original image is eroded, that is, "the domain is eroded", and the image obtained through the forward edge processing operation has a smaller height than the original image. Bright area. In some embodiments, the reverse edge processing operation may be an expansion operation performed on the highlighted part in the image, that is, "domain expansion", and the image obtained by the reverse edge processing operation has a larger size than the original image. Highlight the area.

In one embodiment, the image boundary acquisition device based on the point cloud map may perform global positive correction on all image areas on the point cloud map when performing edge processing operations on the image areas with continuous same semantics To the edge processing operation, determine the image boundary of the pseudo-adhesion, so as to divide the image regions of the pseudo-adhesion; and / or, perform the local positive edge processing operation on the image regions connected on the point cloud map, The semi-adhesive image boundary is determined to divide the semi-adhesive image area among the connected image areas. By performing edge processing on the image of the point cloud map, the pseudo-adhesive and semi-adhesive regions can be segmented and the overlapping regions can be segmented, which improves the accuracy of segmenting the image regions.

For example, assuming that the point cloud map is a point cloud map of Daejeon, the image boundary acquisition device based on the point cloud map may perform a global positive edge processing operation on all image areas on the point cloud map to determine the false The image boundary of adhesion is to divide each image area of pseudo adhesion. The image boundary acquisition device based on the point cloud map may also determine the image areas connected on the point cloud map according to the semantics of the point cloud map, and perform the image areas connected on the point cloud map. The local positive edge processing operation determines the semi-adhesive image boundary, so as to segment the semi-adhesive image region among the connected image regions. After performing the corrosion operation on the point cloud map, the image boundary acquisition device based on the point cloud map may also perform a reverse edge processing operation on the point cloud map, thereby dividing the field into multiple images with different semantics region.

For details, please refer to FIG. 1. FIG. 1 is a schematic diagram of a working scene of an image boundary acquisition system based on a point cloud map provided by an embodiment of the present invention. The image boundary acquisition system based on a point cloud map shown in FIG. 1 includes: An image boundary acquisition device 11 for a cloud map and an aircraft 12, the image boundary acquisition device 11 based on a point cloud map may be a control terminal of the aircraft 12, specifically a remote controller, a smartphone, a tablet computer, a laptop computer, Any one or more of ground stations and wearable devices (watches, bracelets). The aircraft 12 may be a rotor-type aircraft, such as a four-rotor aircraft, a six-rotor aircraft, an eight-rotor aircraft, or a fixed-wing aircraft. The aircraft 12 includes a power system 121 for providing flight power to the aircraft 12, wherein the power system 121 includes any one or more of a propeller, a motor, and an electronic governor. The aircraft 12 may further include a pan / tilt 122 and The imaging device 123 is mounted on the main body of the aircraft 12 via the gimbal 122. The camera device 123 is used for taking images or videos during the flight of the aircraft 12, including but not limited to multi-spectral imagers, hyper-spectral imagers, visible light cameras and infrared cameras, etc. The gimbal 122 is a multi-axis transmission and stabilization system The PTZ 122 motor compensates the imaging angle of the imaging device by adjusting the rotation angle of the rotation axis, and prevents or reduces the shaking of the imaging device by setting an appropriate buffer mechanism.

In the embodiment of the present invention, the point cloud map-based image boundary acquisition system may acquire a point cloud map containing semantics through the point cloud map-based image boundary acquisition device 11, and according to the semantics on the point cloud map, Each image area with different semantics on the point cloud map is determined.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of a method for acquiring an image boundary based on a point cloud map according to an embodiment of the present invention. The method may be performed by an image boundary acquiring device based on a point cloud map. The specific explanation of the image boundary acquisition device of the point cloud map is as described above. Specifically, the method in the embodiment of the present invention includes the following steps.

S201: Obtain a point cloud map containing semantics.

In an embodiment of the present invention, an image boundary acquisition device based on a point cloud map can acquire a point cloud map containing semantics. In some embodiments, the point cloud map is generated according to the semantics of each pixel on the image captured by the camera. In some embodiments, the point cloud map contains a plurality of point data, and each point data includes location data, altitude data, and multiple semantics with different confidence levels.

In one embodiment, before acquiring the point cloud map based on the point cloud map, the image boundary acquisition device may collect sample image data through the camera of the aircraft, and perform a sample image corresponding to the sample image data. Semantic annotation, obtaining sample image data including semantic annotation information, and generating an initial semantic recognition model according to a preset semantic recognition algorithm, so that the sample image data including semantic annotation information is used as input data and input into the initial semantic recognition model Train to generate a semantic recognition model.

In some embodiments, the sample image data may include a color image or an orthophoto; or, the sample image may include a color image and depth of field data corresponding to the color image; or, the sample image may include an orthophoto Depth of field data corresponding to the image and the orthophoto. In some embodiments, the orthophoto is an aerial image that has been geometrically corrected (for example, to have a uniform scale). Unlike the aerial image that has not been corrected, the amount of orthophoto can be used to measure the actual Distance, because it is a true description of the earth's surface obtained through geometric correction, the orthophotos have the characteristics of being rich in information, intuitive and measurable. In some embodiments, the color image is an image determined according to RGB values. In some embodiments, the depth of field data reflects the distance from the camera to the object.

In one embodiment, when acquiring a point cloud map based on a point cloud map, the image boundary acquisition device may acquire the first image data collected by a camera mounted on the aircraft during the flight of the aircraft , And input the first image data into the semantic recognition model for processing, identify the semantics of each pixel in the first image data, and according to the identified corresponding to the first image data Position data, height data, and the semantics of each pixel in the first image data generate first point cloud data containing semantics, thereby generating a point cloud map using the first point cloud data containing semantics.

In one embodiment, the semantic recognition model used in this solution may be a Convolutional Neural Network (CNN) model. The architecture of the CNN model mainly includes an input layer, a convolutional layer, an excitation layer, and pooling Floor. In the neural network model, a plurality of subnets may be included, the subnets are arranged in a sequence from lowest to highest, and the input image data is processed by each of the subnets in the sequence. The subnets in the sequence include multiple module subnets and optionally one or more other subnets, all of which are composed of one or more conventional neural network layers, such as maximum pooling layer, convolutional layer , Fully connected layer, regularization layer, etc. Each subnet receives the previous output representation generated by the previous subnet in the sequence; processes the previous output representation by pass-through convolution to generate a pass-through output; and processes it by one or more groups of neural network layers. The front output representation is used to generate one or more groups, and the through output and the group output are connected to generate an output representation of the module subnet.

In some embodiments, the input layer is used to input image data, the convolution layer is used to perform operations on the image data, and the excitation layer is used to perform non-linear mapping on the output of the convolution layer. The pooling layer is used to compress the amount of data and parameters, reduce overfitting, and improve performance. This solution uses the sample image data after semantic annotation as input data, enters the input layer of the CNN model, and after the calculation of the convolution layer, outputs the confidence of different semantics through multiple channels, for example, farm channel (confidence), fruit tree Channel (confidence), river channel (confidence), etc. As the output result of CNN, it can be expressed as a tensor value. For example, for a certain pixel {longitude, latitude, height, K1, K2, ..., Kn}, the tensor value represents the three-dimensional point cloud information of the pixel and n The semantic information of the channel, where K1, K2, ..., Kn represent the confidence, and the semantic channel with the highest confidence in the tensor data is taken as the semantics of the pixel. For example, if the confidence of the i-th semantic channel is Ki = 0.8, which is the highest confidence, then the semantics corresponding to the i-th channel are taken as the semantics of the pixel.

S202: Determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map.

In the embodiment of the present invention, the image boundary acquisition device based on the point cloud map may determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map.

In one embodiment, the image boundary acquisition device based on the point cloud map may determine the image regions of different semantics on the point cloud map according to the semantics on the point cloud map, according to the point cloud map On the point cloud map, determine image areas with continuous and identical semantics on the point cloud map, and perform edge processing operations on the image areas with continuous and identical semantics to obtain image areas with different semantics on the point cloud map . In some embodiments, the edge processing operation includes a forward edge processing operation and / or a reverse edge processing operation. In some embodiments, the forward edge processing operation may include an erosion operation, and the reverse edge processing operation may include an expansion operation. In some embodiments, the formula of the corrosion operation is shown in formula (1):

dst (x, y) = min src (x + x ', y + y')

(x ', y'): element (x ', y') ≠ 0 (1)

Among them, in the above formula (1), dst (x, y) represents the target pixel value of the corrosion operation, (x, y), (x ', y') represents the pixel coordinate position, src (x + x ', y + y ') means value operation.

In some embodiments, the formula of the expansion operation is shown in formula (2):

dst (x, y) = max src (x + x ', y + y')

(x ', y'): element (x ', y') ≠ 0 (2)

Among them, in the above formula (2), dst (x, y) represents the target pixel value of the expansion operation, (x, y), (x ', y') represents the pixel coordinate position, src (x + x ', y + y ') means value operation.

In one embodiment, the positive edge processing operation includes: performing a global positive edge processing operation on all image areas on the point cloud map to determine the image boundary of the pseudo-adhesion, Segment each image area; and / or, perform a local positive edge processing operation on each image area connected on the point cloud map to determine a semi-adhesive image boundary, so as to The semi-adhesive image area is segmented.

In one embodiment, the global positive edge processing operation includes: convolving each semantic set image in the point cloud map with a preset computing kernel to obtain the pixel of the area covered by the computing kernel The minimum value, and assign the minimum value to the specified pixel. In some embodiments, the local positive edge processing operation includes: convolving the semantic collection image with connected domains in the point cloud map with a preset calculation kernel to obtain pixels of the area covered by the calculation kernel The minimum value of the point, and assign the minimum value to the specified pixel. In some embodiments, the preset calculation kernel is a predetermined figure with reference points.

Specifically, FIG. 3.1 can be used as an example for illustration, and FIG. 3.1 is a schematic diagram of an etching operation provided by an embodiment of the present invention. As shown in FIG. 3.1, assuming that the image area of the point cloud map is the semantic collection image 311, the image boundary acquisition device based on the point cloud map may use each semantic collection image 311 in the point cloud map as The predetermined figure 312 with reference points of the preset calculation kernel is convoluted to obtain the minimum value of the pixels of the area covered by the calculation kernel, and the minimum value is assigned to the specified pixel, as shown in Figure 3.1 Of the corrosion image 313.

In some embodiments, the reverse edge processing operation includes: convolving each semantic set image in the point cloud map with a preset calculation kernel to obtain the maximum value of the pixels of the area covered by the calculation kernel And assign the maximum value to the specified pixel. In some embodiments, the preset calculation kernel is a predetermined figure with reference points.

Specifically, FIG. 3.2 can be used as an example for illustration, and FIG. 3.2 is a schematic diagram of an expansion operation provided by an embodiment of the present invention. As shown in FIG. 3.2, assuming that the image area of the point cloud map is the semantic collection image 321, the image boundary acquisition device based on the point cloud map may use each semantic collection image 321 in the point cloud map as The predetermined graph 322 with reference points of the preset calculation kernel is convoluted to obtain the maximum value of the pixels of the area covered by the calculation kernel, and the maximum value is assigned to the specified pixel, and the minimum The value is assigned to the specified pixel, and the expanded image 323 shown in Figure 3.2 is obtained.

Through the forward edge processing operation, a highlight area smaller than the original image can be obtained, and through the reverse edge processing operation, a highlight area larger than the original image can be obtained. Through this embodiment, the image effect can be enhanced, and more effective data can be provided for the calculation in the subsequent image processing process, so as to improve the accuracy of the calculation.

In the embodiment of the present invention, an image boundary acquisition device based on a point cloud map may acquire a point cloud map containing semantics, and determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map, by In this way, the image area can be automatically divided, which meets the needs of automation and intelligence to classify the image area, and improves the accuracy of image division.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of a route planning method based on a point cloud map provided by an embodiment of the present invention. The method may be executed by a route planning device based on a point cloud map. The route planning equipment of the map can be installed on the aircraft, or on other mobile equipment that establishes a communication connection with the aircraft, such as autonomous equipment such as robots, unmanned vehicles, and unmanned boats. In some embodiments, the point cloud map-based route planning device may be a component of an aircraft; in other embodiments, the point cloud map-based route planning device may also be spatially independent of the aircraft. Specifically, the method in the embodiment of the present invention includes the following steps.

S401: Obtain a point cloud map containing semantics.

In an embodiment of the present invention, a route planning device based on a point cloud map can obtain a point cloud map containing semantics.

In one embodiment, when acquiring a point cloud map containing semantics, a route planning device based on a point cloud map may acquire first image data captured by a camera device mounted on the aircraft, and process the first image data based on a semantic recognition model Image data to obtain the semantics of each pixel in the first image data, and the position data, height data corresponding to the first image data and each pixel in the first image data To generate the first point cloud data containing semantics, so as to generate a point cloud map using the first point cloud data containing semantics.

In one embodiment, the route planning device based on the point cloud map may train and generate the semantic recognition model before processing the first image data based on the semantic recognition model. When training to generate the semantic recognition model, the point cloud map-based route planning device may collect sample image data through the camera of the aircraft, and semantically annotate the sample image corresponding to the sample image data to obtain including semantic annotation Sample image data for information. The route planning device based on the point cloud map may generate an initial semantic recognition model according to a preset semantic recognition algorithm, and use the sample image data including semantic annotation information as input data, input the initial semantic recognition model for training, A training result is obtained, where the training result includes position data corresponding to the sample image data, height data, and the semantics of each pixel in the sample image. In some embodiments, the position data corresponding to the sample image data includes the longitude and latitude of the sample image, and the height data corresponding to the sample image data is the height of the sample image. After obtaining the training result, the route planning device based on the point cloud map may compare the semantics of each pixel in the sample image in the training result with the semantic annotation information of the sample image, if it does not match, it may Adjusting the parameters in the initial semantic recognition model until the semantics of each pixel in the training result sample image matches the semantic annotation information, the semantic recognition model is generated.

In some embodiments, the first point cloud data corresponds to each pixel in the first image data, and the semantics of different point cloud data on the point cloud map can be marked with different display methods, Such as marking by different colors. As shown in FIG. 5, FIG. 5 is a schematic diagram of an interface of a point cloud map provided by an embodiment of the present invention. FIG. 5 is a schematic diagram of tagging point cloud data with different semantics on a point cloud map by using different colors. FIG. 5 The different colors shown in represent different categories.

In one embodiment, when the first image data includes orthophotos, the route planning device based on the point cloud map may semantically label the orthophotos (that is, mark the categories of features, so that Recognize feature types), obtain orthophotos containing semantic annotation information, and input the orthophotos containing semantic annotation information into the trained semantic recognition model for processing, and identify the orthophotos on the orthophotos Semantics corresponding to each pixel, and output semantic confidence, position data and height data of each pixel on the orthophoto. In some embodiments, the position data includes the longitude and latitude of the first image in the first image data, and the height data includes the height of the first image in the first image data.

In one embodiment, when the first image data includes an orthophoto and depth of field data corresponding to the orthophoto, the point cloud map-based route planning device may use a trained semantic recognition model to The orthophoto and the depth data corresponding to the orthophoto are identified, and the semantics corresponding to each pixel on the orthophoto are identified. The route planning device based on the point cloud map may generate a first point cloud containing semantics according to the position data, altitude data, depth data corresponding to the orthophoto and the semantics corresponding to each pixel on the orthophoto Data to generate a point cloud map containing semantics. In some embodiments, the depth of field data may be displayed by a depth map. The depth map refers to a frame of data with depth information (that is, depth of field data) read from the camera device. It is suitable for intuitive viewing, so the depth map can be converted into point cloud data according to preset rules, so that a point cloud map can be generated according to the point cloud data, which is convenient for users to view.

In some embodiments, the first image data includes orthophotos. Since the orthophotos obtained at different times may have a large overlap, the two orthophotos collected at two different times may be There may be multiple pixels with the same position data, and the semantics of the identified multiple pixels with the same position data in the two orthophotos may be inconsistent. Therefore, in order to more reliably perform semantic recognition on multiple pixels with the same location data, the route planning device based on the point cloud map can output the semantic confidence of the semantics of the multiple pixels with the same location data according to the semantic recognition model To determine the semantics with higher confidence as the semantics of multiple pixels with the same position data.

In some embodiments, the point cloud map-based route planning device may also use manual voting to determine the semantics of multiple pixels with the same location data; in some embodiments, the point cloud map-based Of the route planning device can also determine the semantics of multiple pixels with the same location data as the most marked times as the semantics of multiple pixels with the same location data; in other embodiments, multiple The semantics of the pixel can also be determined according to other rules, for example, according to the preset semantic priority, which is not specifically limited in this embodiment of the present invention.

In one embodiment, the semantic recognition model used in this solution may be a CNN model, and the architecture of the CNN model mainly includes an input layer, a convolutional layer, an excitation layer, and a pooling layer. In the neural network model, a plurality of subnets may be included, the subnets are arranged in a sequence from lowest to highest, and the input image data is processed by each of the subnets in the sequence. The subnets in the sequence include multiple module subnets and optionally one or more other subnets, all of which are composed of one or more conventional neural network layers, such as maximum pooling layer, convolutional layer , Fully connected layer, regularization layer, etc. Each subnet receives the previous output representation generated by the previous subnet in the sequence; processes the previous output representation by pass-through convolution to generate a pass-through output; and processes it by one or more groups of neural network layers. The front output representation is used to generate one or more groups, and the through output and the group output are connected to generate an output representation of the module subnet.

In some embodiments, the input layer is used to input image data, the convolution layer is used to perform operations on the image data, and the excitation layer is used to perform non-linear mapping on the output of the convolution layer. The pooling layer is used to compress the amount of data and parameters, reduce overfitting, and improve performance. This solution uses the sample image data after semantic annotation as input data, input to the input layer of the CNN model, and after the calculation of the convolutional layer, the confidence of different semantics is output through multiple channels. Specific embodiments are exemplified above, and will not be repeated here.

In one embodiment, the position data includes longitude and latitude; the first point cloud data includes a plurality of point data, and each point data includes position data, height data, and multiple semantics with different confidence levels, and the Each point data contained in the first point cloud data corresponds to each pixel point in the first image data. In some embodiments, the multiple semantics with different confidence levels are obtained from multiple channels after being recognized by the semantic recognition model; in some embodiments, the difference from the output of the general neural network is that A segmented output function is added after the output channel of the neural network. If the channel confidence result is negative, the channel confidence result is set to zero to ensure that the neural network output confidence is positive floating-point data. Using positive floating-point data as the confidence level of the semantic channel, you can directly obtain greater confidence through the subtraction operation of the two pixel data. Since the subtraction operation of the tensor only needs to perform subtraction operations on the numerical content corresponding to the array The amount is very small, and the calculation speed can be greatly improved under the same computing power. Especially suitable for the process of high-precision map drawing, because the high-precision map requires a large amount of calculation, which causes the problem of computing power shortage.

In an embodiment, a route planning device based on a point cloud map may acquire second image data captured by a camera mounted on an aircraft, and process the second image data based on the semantic recognition model to obtain the first The semantics of each pixel in the second image data, and according to the position data, height data corresponding to the second image data and the semantics of each pixel in the second image data, a Two point cloud data, thereby updating the point cloud map using the second point cloud data.

In one embodiment, the first point cloud data, the second point cloud data, and the point cloud map all contain a plurality of point data, and each point data includes position data, altitude data, and multiple semantics with different confidence levels Each point data contained in the first point cloud data corresponds to each pixel in the first image data, and each point data contained in the second point cloud data corresponds to the second image data Corresponds to each pixel. In some embodiments, the confidence level is positive floating point data.

In one embodiment, before updating the point cloud map, the route planning device based on the point cloud map may detect whether the second point cloud exists in the point cloud map generated from the first point cloud data Point data where the data has the same position data (ie overlapping pixels); if it is detected that there is point data with the same position data as the second point cloud data in the point cloud map generated from the first point cloud data , You can compare the semantic confidence of two point data with the same position data in the second point cloud data and the point cloud map, and retain the point data with higher confidence in the two point data Semantic.

In one embodiment, when the point cloud map-based route planning device uses the second point cloud data to update the point cloud map, the two point data may have higher confidence point data The semantics of is determined as the semantics of point data in the point cloud map that is the same as the position data of the second point data, and the point data in the second point cloud data that is different from the position data in the point cloud map Overlay with the point cloud map, so as to update the point cloud map.

In some embodiments, two point data having the same position data in the first point cloud data and the second point cloud data overlap two of the first image data and the second image data Pixels correspond.

In one embodiment, when comparing the second point cloud data and the two point data with the same position data in the point cloud map, the route planning device based on the point cloud map may compare the first point cloud A plurality of semantics of different confidence levels in two point data with the same position data in the data and the second point cloud data are subtracted. In some embodiments, the subtraction operation is to remove the semantics with lower confidence in the two point data and retain the semantics with higher confidence.

For example, it is assumed that the route planning device based on the point cloud map detects that the point cloud map generated from the first point cloud data has the same position data as the second point cloud data before updating the point cloud map Point data, if the semantics of the point data of the same location data in the point cloud map generated from the first point cloud data are fruit trees, and the confidence level is 50%, and the second point cloud data The semantic of the point data of the same position data is rice, and the confidence is 80%, then the semantic confidence of the two point data with the same position data in the second point cloud data and the point cloud map can be compared Since the confidence level of 80% is greater than 50%, the semantics that are lower in the two point data, that is, fruit trees, can be removed, and the semantics in the point cloud map can be updated to rice.

In one embodiment, when the point cloud map-based route planning device uses the second point cloud data to update the point cloud map, the point cloud map generated from the first point cloud data may also be calculated Neutralize the number of semantics of the two point data with the same position data in the second point cloud data in the history records, and use the largest number of semantics as the first point cloud data and all The semantics of the two point data with the same position data in the second point cloud data are described.

In one embodiment, when the point cloud map-based route planning device uses the second point cloud data to update the point cloud map, it may also be based on the second point cloud data and the first point Priority corresponding to the semantics of the two point data with the same position data in the point cloud map generated by the cloud data, and determining the semantics with the highest priority is that the second point cloud data and the position data in the point cloud map are the same The semantics of the two point data.

S402: Determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map.

In the embodiment of the present invention, the route planning device based on the point cloud map may determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map. In some embodiments, each image area included in the point cloud map is divided according to the semantics of each pixel in the point cloud map, and each image area may be displayed by different display marking methods, for example, by Different colors mark each image area with different semantics. Specific embodiments are as described above, and will not be repeated here.

S403: Plan a flight route according to the semantics of each image area on the point cloud map.

In the embodiment of the present invention, the route planning device based on the point cloud map may plan the flight route according to the semantics of each image area on the point cloud map.

In one embodiment, after the point cloud map-based route planning device generates a point cloud map, the flight route can be planned according to the semantics of pixel points corresponding to each image area on the point cloud map. The route planning device based on the point cloud map may determine the obstacle area on the point cloud map according to the semantics of pixel points corresponding to each image area on the point cloud map, and pass the obstacle area through a specific marking method Automatic marking, for example, telephone poles in farmland, isolated trees in farmland, etc. After the obstacle area is automatically marked, the route planning device based on the point cloud map can generate a flight route that automatically avoids the marked obstacle area according to a preset route generation algorithm.

Through this implementation of route planning based on semantic point cloud images, the areas corresponding to the semantics designated as obstacles or obstacle areas can be automatically marked as obstacle areas to be avoided by the route, which is greatly reduced To reduce the workload of relying on manual interpretation of obstacles; by updating the point cloud map containing semantics in real time, the point cloud map merges the results of recognition in multiple orthophotos, reducing the misjudgment or omission of ground features Probability improves the efficiency of identifying features.

Specifically, it can be illustrated with reference to Figures 6.1, 6.2, and 6.3. Figure 6.1 is a schematic diagram of an orthophoto image interface provided by an embodiment of the present invention, and Figure 6.2 is another interface of a point cloud map provided by an embodiment of the present invention. Schematic diagram, FIG. 6.3 is a schematic diagram of an interface of a point cloud map for marking obstacles provided by an embodiment of the present invention. The image boundary acquisition device based on the point cloud map can input the orthophoto shown in FIG. 6.1 into the trained semantic recognition model according to the acquired orthophoto shown in FIG. 6.1, and recognize the image shown in FIG. 6.1 The semantics of the pixels corresponding to the orthophoto. Since different semantics correspond to different types of features, assuming that different semantics are represented by different colors, and each color represents a type of feature, the point cloud map-based image boundary acquisition device The point cloud map is rendered to obtain the point cloud map shown in FIG. 6.2, where the gray dots in the area 601 in FIG. 6.2 represent obstacles such as telephone poles that need to be marked. Therefore, by marking the gray dots in the area 601 in FIG. 6.2, such as marking the gray dots in the area 601 with the circle shown in FIG. 6.3, a schematic diagram of the marked obstacle as shown in FIG. 6.3 can be obtained . In other embodiments, the marking method for the obstacle may be other marking methods, which is not specifically limited in the embodiment of the present invention.

In one embodiment, the route planning device based on the point cloud map may divide the categories of aerial photography scenes based on image regions with different semantics. When the route planning device based on the point cloud map divides the category of the aerial photography scene, the aerial photography scene can be based on the semantic confidence, position data, and altitude data corresponding to each pixel in the point cloud map. To classify.

Specifically, for example, assuming that the aerial scene is a field, and the categories in the field include trees, roads, ground, telephone poles, buildings, water surface, rice fields, other crops, etc., the route based on the point cloud map The planning device may determine, according to any one or more of semantic confidence, position data, and height data corresponding to each pixel point of the point cloud map, pixels whose semantics are trees and whose height data is greater than a first preset height threshold The area corresponding to the point is the area of the tree; the area corresponding to the pixel point whose semantic meaning is cement and / or asphalt is the road; the pixel position corresponding to the semantic confidence level is cement and asphalt is the road; the semantic meaning is the rod, And the area corresponding to the pixels whose height data is greater than the second preset height threshold is a telephone pole; it is determined that the area corresponding to the pixels covered by water such as water and rivers is the water surface; (Excluding water surface), factory buildings, plastic sheds, etc. are buildings; areas corresponding to pixels whose semantic meaning is rice are determined as paddy fields; pixels whose blank area or other semantics whose height data is less than the third preset height threshold are determined The corresponding area is the ground. According to the identified categories included in the field, the areas corresponding to the field are divided.

In one embodiment, the point cloud map containing semantics can also be applied to the detection of illegal buildings, and the route planning device based on the point cloud map can be based on orthophotos with semantic annotation information (ie, first image data ), Through the semantic recognition model to identify the semantics of the pixels corresponding to the two orthophotos collected at different times, and according to the position data, height data and the semantics of each pixel corresponding to the orthophotos collected at two different times, Generate point cloud data with semantics and use point cloud data to generate point cloud maps with semantics. If two pixels with the same location data are detected on two point cloud maps, the semantic confidence of the pixels with the same location data (that is, feature category) can be compared to determine the pixels with the same location data Semantics, so as to determine whether there is illegal building in the pixel area with the same position data according to the semantics; or whether the pixel area with the same position data has changed. Through the implementation of a point cloud map with semantics, it is possible to more reliably detect the change area and provide more detailed change information.

In one embodiment, the point cloud map containing semantics can also be applied to feature classification. Specifically, the features on the point cloud map may be classified according to the semantics of the corresponding pixel points on the point cloud map, the position data and height data of the corresponding pixel points on the point cloud map, and / or the The features on the point cloud map are divided or divided by category.

In one embodiment, the point cloud map containing semantics can also be applied to agricultural machinery spraying tasks. For the planning of flight routes of agricultural machinery spraying tasks, pesticide spraying can be controlled by judging whether the area where the agricultural machinery is flying is a crop that needs to be sprayed Switch to avoid wasting pesticides.

S404: Control the aircraft to fly according to the flight path.

In the embodiment of the present invention, a route planning device based on a point cloud map may control the aircraft to fly according to the flight route.

In one embodiment, when the route planning device based on the point cloud map controls the aircraft to fly according to the flight route, it can determine the semantics of the image area corresponding to the current flight position of the aircraft in the point cloud map Whether it matches the semantics of the target mission, if it is determined that the semantics of the image area corresponding to the current flight position of the aircraft in the point cloud map match the semantics of the target mission, the aircraft can be controlled to execute the Target mission; if it is determined that the semantics of the image area corresponding to the current flight position of the aircraft in the point cloud map do not match the semantics of the target mission, the aircraft can be controlled to stop performing the target mission. In some embodiments, the target task may be any one or more tasks such as a pesticide spraying task, an obstacle detection task, and classifying scene targets.

In one embodiment, if the target task is to classify scene targets, the route planning device based on the point cloud map may identify the targets of the aerial scene when controlling the aircraft to perform the target tasks, And generate a point cloud map containing semantics according to the recognition result, and classify the aerial photography scene according to the point cloud map containing semantics.

In the embodiment of the present invention, a route planning device based on a point cloud map may obtain a point cloud map containing semantics, and determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map, and The semantic route of each image area on the point cloud map is used to plan a flight route, thereby controlling the aircraft to fly according to the flight route. Through this implementation manner, it is possible to plan flight routes according to different semantics to avoid obstacle areas and improve flight safety of the aircraft.

Please refer to FIG. 7, which is a schematic structural diagram of an image boundary acquisition device based on a point cloud map according to an embodiment of the present invention. Specifically, the image boundary acquisition device based on the point cloud map includes: a memory 701, a processor 702, and a data interface 703.

The memory 701 may include a volatile memory (volatile memory); the memory 701 may also include a non-volatile memory (non-volatile memory); the memory 701 may also include a combination of the foregoing types of memories. The processor 702 may be a central processing unit (central processing unit, CPU). The processor 702 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. For example, it may be a complex programmable logic device (complex programmable logic device, CPLD), field programmable logic gate array (field-programmable gate array, FPGA), or any combination thereof.

Further, the memory 701 is used to store program instructions. When the program instructions are executed, the processor 702 may call the program instructions stored in the memory 701 to perform the following steps:

Get a point cloud map with semantics;

Further, when the processor 702 determines each image area with different semantics on the point cloud map according to the semantics on the point cloud map, it is specifically used to:

According to the semantics on the point cloud map, determine an image area on the point cloud map that has continuous and same semantics;

Perform an edge processing operation on each image region having the same continuous semantics to obtain each image region with different semantics on the point cloud map.

Further, the edge processing operation includes: a forward edge processing operation and / or a reverse edge processing operation.

Further, the forward edge processing operation includes:

Perform global positive edge processing on all image areas on the point cloud map to determine the image boundary of pseudo-adhesion, so as to segment each image area of pseudo-adhesion; and / or,

Perform a local positive edge processing operation on each connected image area on the point cloud map to determine a semi-adhesive image boundary, so as to divide the semi-adhesive image area among the connected image areas.

Further, the global edge processing operation includes:

Each semantic collection image in the point cloud map is convolved with a preset calculation kernel to obtain the minimum value of the pixels in the area covered by the calculation kernel, and the minimum value is assigned to the specified pixel.

Further, the local positive edge processing operation includes:

Convolution of the semantic collection image with connected domains in the point cloud map with a preset calculation kernel to obtain the minimum value of the pixels of the area covered by the calculation kernel, and assign the minimum value to the specified pixel point.

Further, the reverse edge processing operation includes:

Each semantic set image in the point cloud map is convoluted with a preset calculation kernel to obtain the maximum value of the pixels in the area covered by the calculation kernel, and the maximum value is assigned to the specified pixel.

Further, the preset calculation kernel is a predetermined figure with reference points.

Please refer to FIG. 8, which is a schematic structural diagram of a route planning device based on a point cloud map according to an embodiment of the present invention. Specifically, the route planning device based on the point cloud map includes: a memory 801, a processor 802, and a data interface 803.

The memory 801 may include a volatile memory (volatile memory); the memory 801 may also include a non-volatile memory (non-volatile memory); the memory 801 may also include a combination of the foregoing types of memories. The processor 802 may be a central processing unit (central processing unit, CPU). The processor 802 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. For example, it may be a complex programmable logic device (complex programmable logic device, CPLD), a field programmable logic gate array (field-programmable gate array, FPGA), or any combination thereof.

Further, the memory 801 is used to store program instructions. When the program instructions are executed, the processor 802 may call the program instructions stored in the memory 801 to perform the following steps:

Get a point cloud map with semantics;

Controlling the aircraft to fly according to the flight path.

Further, when the processor 802 obtains a point cloud map containing semantics, it is specifically used to:

Obtain the first image data captured by the camera device mounted on the aircraft;

Processing the first image data based on a semantic recognition model to obtain the semantics of each pixel in the first image data;

Generating first point cloud data containing semantics according to the position data, height data corresponding to the first image data, and the semantics of each pixel in the first image data;

A point cloud map is generated using the first point cloud data containing semantics.

Obtain the second image data captured by the camera device mounted on the aircraft;

Processing the second image data based on the semantic recognition model to obtain the semantics of each pixel in the second image data;

Generate second point cloud data containing semantics according to the position data, height data corresponding to the second image data, and the semantics of each pixel in the second image data;

Update the point cloud map using the second point cloud data.

Further, the first point cloud data, the second point cloud data, and the point cloud map all contain a plurality of point data, and each point data includes position data, height data, and multiple semantics with different confidence levels;

Each point data included in the first point cloud data corresponds to each pixel in the first image data, and each point data included in the second point cloud data corresponds to the Each pixel corresponds.

Further, the confidence level is positive floating point data.

Further, when the processor 802 uses the second point cloud data to update the point cloud map, it is specifically used to:

Compare two point data with the same position data in the second point cloud data and the point cloud map, and retain the point data with higher confidence in the two point data.

Further, when the processor 802 compares the second point cloud data and the two point data with the same position data in the point cloud map, it is specifically used to:

Subtraction operations are performed on a plurality of semantics with different confidence levels in two point data with the same position data in the first point cloud data and the second point cloud data.

Further, two point data having the same position data in the first point cloud data and the second point cloud data correspond to two overlapping pixel points in the first image data and the second image data.

Count the number of semantics of the two point data with the same position data in the first point cloud data and the second point cloud data are marked as the number of the same semantics in the history record;

The semantics with the largest number is used as the semantics of the two point data with the same position data in the first point cloud data and the second point cloud data.

According to the priorities corresponding to the semantics of the two point data with the same position data in the second point cloud data and the point cloud map, it is determined that the semantics with the highest priority are the second point cloud data and the The semantics of two point data with the same position data in a point cloud map.

Further, the first image data includes a color image; or,

The first image data includes a color image and depth data corresponding to the color image; or,

The first image data includes an orthophoto; or,

The first image data includes orthophotos and depth data corresponding to the orthophotos.

Further, before processing the first image data based on the semantic recognition model, the processor 802 is further used to:

Acquiring a sample database, the sample database including sample image data;

Generate an initial semantic recognition model according to a preset semantic recognition algorithm;

Training and optimizing the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model;

Wherein, the sample image data includes a sample image and semantic annotation information; or, the sample image data includes a sample image, depth data corresponding to each pixel in the sample image and semantic annotation information.

Further, when the processor 802 trains and optimizes the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model, it is specifically used to:

Calling the initial semantic recognition model to identify the sample image included in the sample image data and the depth data corresponding to each pixel in the sample image to obtain a recognition result;

If the recognition result matches the semantic annotation information included in the sample image data, the model parameters of the initial semantic recognition model are optimized to obtain the semantic recognition model.

Further, the point cloud map includes a plurality of image areas, the image areas are divided according to the semantics of each pixel in the point cloud map, and each image area is displayed by different display mark methods.

Further, the processor 802 is specifically used when planning a flight route according to the semantics of each image area on the point cloud map:

Determine the obstacle area on the point cloud map according to the semantics corresponding to each image area on the point cloud map;

When planning the route, bypass the obstacle area to plan the flight route.

Further, when the processor 802 controls the aircraft to fly according to the flight path, it is specifically used to:

In the process of controlling the aircraft to fly according to the flight path, determine whether the semantics of the image area corresponding to the current flying position of the aircraft in the point cloud map match the semantics of the target task;

If the judgment result is yes, control the aircraft to perform the target mission;

If the judgment result is no, the aircraft is controlled to stop performing the target mission.

An embodiment of the present invention provides an aircraft including: a fuselage; a power system provided on the fuselage for providing flight power; the power system includes: a blade and a motor for driving the blade to rotate; The processor is used to obtain a point cloud map containing semantics; according to the semantics on the point cloud map, determine each image area with different semantics on the point cloud map.

Further, when the processor determines each image area with different semantics on the point cloud map according to the semantics on the point cloud map, it is specifically used to:

Further, the positive edge processing operation includes:

Further, the global edge processing operation includes:

Further, the local positive edge processing operation includes:

Further, the reverse edge processing operation includes:

In the embodiment of the present invention, an image boundary acquisition device based on a point cloud map may acquire a point cloud map containing semantics, and determine each image area with different semantics on the point cloud map according to the semantics on the point cloud map, by In this way, the image area can be automatically divided, which meets the needs of automation and intelligence for the classification of the image area, and improves the accuracy of image division.

An embodiment of the present invention also provides an aircraft including: a fuselage; a power system provided on the fuselage for providing flight power; the power system includes: a blade and a motor for driving the blade to rotate A processor for acquiring a point cloud map containing semantics; determining each image area with different semantics on the point cloud map according to the semantics on the point cloud map; according to the semantics of each image area on the point cloud map , Plan a flight route; control the aircraft to fly according to the flight route.

Further, when the processor obtains a point cloud map containing semantics, it is specifically used to:

Further, the processor is also used to:

Update the point cloud map using the second point cloud data.

Further, the confidence level is positive floating point data.

Further, when the processor uses the second point cloud data to update the point cloud map, it is specifically used to:

Further, when the processor compares the second point cloud data and the two point data with the same position data in the point cloud map, it is specifically used to:

Further, the first image data includes a color image; or,

The first image data includes an orthophoto; or,

Further, before processing the first image data based on the semantic recognition model, the processor is further configured to:

Acquiring a sample database, the sample database including sample image data;

Further, when the processor performs training optimization on the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model, it is specifically used to:

Further, the processor is specifically used when planning a flight route according to the semantics of each image area on the point cloud map:

When planning the route, bypass the obstacle area to plan the flight route.

Further, when the processor controls the aircraft to fly according to the flight path, it is specifically used to:

In an embodiment of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the invention described in the embodiment corresponding to FIG. 2. The method of acquiring the image boundary based on the point cloud map or the route planning method based on the point cloud map described in the embodiment corresponding to FIG. 3 can also realize the method based on the point cloud map of the embodiment corresponding to the present invention shown in FIG. The image boundary acquisition device or the point cloud map-based route planning device according to the embodiment of the present invention described in FIG. 7 will not be repeated here.

The computer-readable storage medium may be an internal storage unit of the device according to any one of the foregoing embodiments, such as a hard disk or a memory of the device. The computer-readable storage medium may also be an external storage device of the device, for example, a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), and a secure digital (SD) card , Flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been or will be output.

A person of ordinary skill in the art may understand that all or part of the processes in the method of the foregoing embodiments may be completed by instructing relevant hardware through a computer program, and the program may be stored in a computer-readable storage medium. During execution, the process of the above method embodiments may be included. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM), etc.

The above disclosure is only part of the embodiments of the present invention, and of course it cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims

An image boundary acquisition method based on a point cloud map, characterized in that the method includes:

Get a point cloud map with semantics;

According to the semantics on the point cloud map, each image area with different semantics on the point cloud map is determined.
The method according to claim 1, wherein the determining each image area with different semantics on the point cloud map according to the semantics on the point cloud map includes:

According to the semantics on the point cloud map, determine an image area on the point cloud map that has continuous and same semantics;

Perform an edge processing operation on each image region having the same continuous semantics to obtain each image region with different semantics on the point cloud map.
The method according to claim 2, wherein the edge processing operation comprises: a forward edge processing operation and / or a reverse edge processing operation.
The method according to claim 3, wherein the positive edge processing operation comprises:

Perform global positive edge processing on all image areas on the point cloud map to determine the image boundary of pseudo-adhesion, so as to segment each image area of pseudo-adhesion; and / or,

Perform a local positive edge processing operation on each connected image area on the point cloud map to determine a semi-adhesive image boundary, so as to divide the semi-adhesive image area among the connected image areas.
The method according to claim 4, wherein the global positive edge processing operation comprises:

Each semantic collection image in the point cloud map is convolved with a preset calculation kernel to obtain the minimum value of the pixels in the area covered by the calculation kernel, and the minimum value is assigned to the specified pixel.
The method according to claim 4, wherein the local positive edge processing operation comprises:

Convolution of the semantic collection image with connected domains in the point cloud map with a preset calculation kernel to obtain the minimum value of the pixels of the area covered by the calculation kernel, and assign the minimum value to the specified pixel point.
The method of claim 3, wherein the reverse edge processing operation comprises:

Each semantic set image in the point cloud map is convoluted with a preset calculation kernel to obtain the maximum value of the pixels in the area covered by the calculation kernel, and the maximum value is assigned to the specified pixel.
The method according to any one of claims 5-7, wherein the preset calculation kernel is a predetermined figure with a reference point.
A route planning method based on a point cloud map, characterized in that the method includes:

Get a point cloud map with semantics;

According to the semantics on the point cloud map, determine each image area with different semantics on the point cloud map;

Plan flight routes according to the semantics of each image area on the point cloud map;

Controlling the aircraft to fly according to the flight path.
The method according to claim 9, wherein the acquiring a point cloud map containing semantics includes:

Obtain the first image data captured by the camera device mounted on the aircraft;

Processing the first image data based on a semantic recognition model to obtain the semantics of each pixel in the first image data;

Generating first point cloud data containing semantics according to the position data, height data corresponding to the first image data, and the semantics of each pixel in the first image data;

A point cloud map is generated using the first point cloud data containing semantics.
The method of claim 10, further comprising:

Obtain the second image data captured by the camera device mounted on the aircraft;

Processing the second image data based on the semantic recognition model to obtain the semantics of each pixel in the second image data;

Generate second point cloud data containing semantics according to the position data, height data corresponding to the second image data, and the semantics of each pixel in the second image data;

Update the point cloud map using the second point cloud data.
The method of claim 11, wherein:

The first point cloud data, the second point cloud data, and the point cloud map all contain a plurality of point data, and each point data includes position data, height data, and multiple semantics with different confidence levels;

Each point data included in the first point cloud data corresponds to each pixel in the first image data, and each point data included in the second point cloud data corresponds to the Each pixel corresponds.
The method of claim 12, wherein the confidence level is positive floating point data.
The method of claim 11, wherein using the second point cloud data to update the point cloud map includes:

Compare two point data with the same position data in the second point cloud data and the point cloud map, and retain the point data with higher confidence in the two point data.
The method according to claim 14, wherein comparing the two point data with the same position data in the second point cloud data and the point cloud map includes:

Subtraction operations are performed on a plurality of semantics with different confidence levels in two point data with the same position data in the first point cloud data and the second point cloud data.
The method according to claim 15, characterized in that

Two point data having the same position data in the first point cloud data and the second point cloud data correspond to two overlapping pixels in the first image data and the second image data.
The method according to claim 14, wherein the updating of the point cloud map using the second point cloud data includes:

Count the number of semantics of the two point data with the same position data in the first point cloud data and the second point cloud data are marked as the number of the same semantics in the history record;

The semantics with the largest number is used as the semantics of the two point data with the same position data in the first point cloud data and the second point cloud data.
The method of claim 14, wherein using the second point cloud data to update the point cloud map includes:

According to the priorities corresponding to the semantics of the two point data with the same position data in the second point cloud data and the point cloud map, it is determined that the semantics with the highest priority are the second point cloud data and the The semantics of two point data with the same position data in a point cloud map.
The method according to claim 10, characterized in that

The first image data includes a color image; or,

The first image data includes a color image and depth data corresponding to the color image; or,

The first image data includes an orthophoto; or,

The first image data includes orthophotos and depth data corresponding to the orthophotos.
The method according to claim 10, wherein before processing the first image data based on the semantic recognition model, comprising:

Acquiring a sample database, the sample database including sample image data;

Generate an initial semantic recognition model according to a preset semantic recognition algorithm;

Training and optimizing the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model;

Wherein, the sample image data includes a sample image and semantic annotation information; or, the sample image data includes a sample image, depth data corresponding to each pixel in the sample image and semantic annotation information.
The method according to claim 20, wherein the training and optimization of the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model includes:

Calling the initial semantic recognition model to identify the sample image included in the sample image data and the depth data corresponding to each pixel in the sample image to obtain a recognition result;

If the recognition result matches the semantic annotation information included in the sample image data, the model parameters of the initial semantic recognition model are optimized to obtain the semantic recognition model.
The method of claim 11, wherein:

The point cloud map includes a plurality of image areas, the image areas are divided according to the semantics of each pixel in the point cloud map, and each image area is displayed by different display mark methods.
The method according to claim 22, wherein the planning of flight routes according to the semantics of each image area on the point cloud map includes:

Determine the obstacle area on the point cloud map according to the semantics of each image area on the point cloud map;

When planning the route, bypass the obstacle area to plan the flight route.
The method according to claim 23, wherein the controlling the aircraft to fly according to the flight path includes:

In the process of controlling the aircraft to fly according to the flight path, determine whether the semantics of the image area corresponding to the current flying position of the aircraft in the point cloud map match the semantics of the target task;

If the judgment result is yes, control the aircraft to perform the target mission;

If the judgment result is no, the aircraft is controlled to stop performing the target mission.
An image boundary acquisition device based on a point cloud map, characterized in that the device includes: a memory and a processor;

The memory is used to store program instructions;

The processor invokes program instructions stored in the memory to perform the following steps:

Get a point cloud map with semantics;

According to the semantics on the point cloud map, each image area with different semantics on the point cloud map is determined.
The device according to claim 25, wherein the processor is specifically used when determining each image area with different semantics on the point cloud map according to the semantics on the point cloud map:

According to the semantics on the point cloud map, determine an image area on the point cloud map that has continuous and same semantics;

Perform an edge processing operation on each image region having the same continuous semantics to obtain each image region with different semantics on the point cloud map.
The apparatus according to claim 26, wherein the edge processing operation comprises a forward edge processing operation and / or a reverse edge processing operation.
The apparatus according to claim 27, wherein the positive edge processing operation comprises:

Perform global positive edge processing on all image areas on the point cloud map to determine the image boundary of pseudo-adhesion, so as to segment each image area of pseudo-adhesion; and / or,

Perform a local positive edge processing operation on each connected image area on the point cloud map to determine a semi-adhesive image boundary, so as to divide the semi-adhesive image area among the connected image areas.
The apparatus according to claim 28, wherein the global positive edge processing operation comprises:

Each semantic collection image in the point cloud map is convolved with a preset calculation kernel to obtain the minimum value of the pixels in the area covered by the calculation kernel, and the minimum value is assigned to the specified pixel.
The apparatus according to claim 28, wherein the local positive edge processing operation comprises:

Convolution of the semantic collection image with connected domains in the point cloud map with a preset calculation kernel to obtain the minimum value of the pixels of the area covered by the calculation kernel, and assign the minimum value to the specified pixel point.
The apparatus according to claim 27, wherein the reverse edge processing operation comprises:

Each semantic set image in the point cloud map is convoluted with a preset calculation kernel to obtain the maximum value of the pixels in the area covered by the calculation kernel, and the maximum value is assigned to the specified pixel.
The device according to any one of claims 29 to 31, wherein the preset calculation kernel is a predetermined figure with a reference point.
A route planning device based on a point cloud map, characterized in that the device includes: a memory and a processor;

The memory is used to store program instructions;

The processor invokes program instructions stored in the memory to perform the following steps:

Get a point cloud map with semantics;

According to the semantics on the point cloud map, determine each image area with different semantics on the point cloud map;

Plan flight routes according to the semantics of each image area on the point cloud map;

Controlling the aircraft to fly according to the flight path.
The device according to claim 33, wherein the processor is specifically used for:

Obtain the first image data captured by the camera device mounted on the aircraft;

Processing the first image data based on a semantic recognition model to obtain the semantics of each pixel in the first image data;

Generating first point cloud data containing semantics according to the position data, height data corresponding to the first image data, and the semantics of each pixel in the first image data;

A point cloud map is generated using the first point cloud data containing semantics.
The device according to claim 34, wherein when the processor obtains a point cloud map containing semantics, it is specifically used to:

Obtain the second image data captured by the camera device mounted on the aircraft;

Processing the second image data based on the semantic recognition model to obtain the semantics of each pixel in the second image data;

Generate second point cloud data containing semantics according to the position data, height data corresponding to the second image data, and the semantics of each pixel in the second image data;

Update the point cloud map using the second point cloud data.
The device according to claim 35, characterized in that

The first point cloud data, the second point cloud data, and the point cloud map all contain a plurality of point data, and each point data includes position data, height data, and multiple semantics with different confidence levels;

Each point data included in the first point cloud data corresponds to each pixel in the first image data, and each point data included in the second point cloud data corresponds to the Each pixel corresponds.
The device of claim 36, wherein the confidence level is positive floating point data.
The device according to claim 35, wherein the processor is specifically configured to: when updating the point cloud map using the second point cloud data:

Compare two point data with the same position data in the second point cloud data and the point cloud map, and retain the point data with higher confidence in the two point data.
The device according to claim 38, wherein the processor is specifically used when comparing the second point cloud data and the two point data with the same position data in the point cloud map:

Subtraction operations are performed on a plurality of semantics with different confidence levels in two point data with the same position data in the first point cloud data and the second point cloud data.
The device according to claim 39, characterized in that

Two point data having the same position data in the first point cloud data and the second point cloud data correspond to two overlapping pixels in the first image data and the second image data.
The device according to claim 38, wherein when the processor uses the second point cloud data to update the point cloud map, the processor is specifically configured to:

Count the number of semantics of the two point data with the same position data in the first point cloud data and the second point cloud data are marked as the number of the same semantics in the history record;

The semantics with the largest number is used as the semantics of the two point data with the same position data in the first point cloud data and the second point cloud data.
The device according to claim 38, wherein when the processor uses the second point cloud data to update the point cloud map, the processor is specifically configured to:

According to the priorities corresponding to the semantics of the two point data with the same position data in the second point cloud data and the point cloud map, it is determined that the semantics with the highest priority are the second point cloud data and the The semantics of two point data with the same position data in a point cloud map.
The device according to claim 34, characterized in that

The first image data includes a color image; or,

The first image data includes a color image and depth data corresponding to the color image; or,

The first image data includes an orthophoto; or,

The first image data includes orthophotos and depth data corresponding to the orthophotos.
The apparatus according to claim 34, wherein the processor is further configured to: before processing the first image data based on the semantic recognition model:

Acquiring a sample database, the sample database including sample image data;

Generate an initial semantic recognition model according to a preset semantic recognition algorithm;

Training and optimizing the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model;

Wherein, the sample image data includes a sample image and semantic annotation information; or, the sample image data includes a sample image, depth data corresponding to each pixel in the sample image and semantic annotation information.
The apparatus according to claim 44, wherein the processor performs training optimization on the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model, specifically to:

Calling the initial semantic recognition model to identify the sample image included in the sample image data and the depth data corresponding to each pixel in the sample image to obtain a recognition result;

If the recognition result matches the semantic annotation information included in the sample image data, the model parameters of the initial semantic recognition model are optimized to obtain the semantic recognition model.
The device according to claim 35, characterized in that

The point cloud map includes a plurality of image areas, the image areas are divided according to the semantics of each pixel in the point cloud map, and each image area is displayed by different display mark methods.
The device according to claim 46, wherein the processor is specifically used when planning a flight route according to the semantics of each image area on the point cloud map:

Determine the obstacle area on the point cloud map according to the semantics corresponding to each image area on the point cloud map;

When planning the route, bypass the obstacle area to plan the flight route.
The apparatus according to claim 47, wherein the processor, when controlling the aircraft to fly according to the flight path, is specifically used to:

In the process of controlling the aircraft to fly according to the flight path, determine whether the semantics of the image area corresponding to the current flying position of the aircraft in the point cloud map match the semantics of the target task;

If the judgment result is yes, control the aircraft to perform the target mission;

If the judgment result is no, the aircraft is controlled to stop performing the target mission.
An aircraft, characterized in that it includes:

body;

A power system provided on the fuselage for providing flight power;

The processor is used to obtain a point cloud map containing semantics; according to the semantics on the point cloud map, determine each image area with different semantics on the point cloud map.
The aircraft according to claim 49, wherein when the processor determines each image area with different semantics on the point cloud map according to the semantics on the point cloud map, it is specifically used to:

According to the semantics on the point cloud map, determine an image area on the point cloud map that has continuous and same semantics;

Perform an edge processing operation on each image region having the same continuous semantics to obtain each image region with different semantics on the point cloud map.
The aircraft according to claim 50, wherein the edge processing operation includes a forward edge processing operation and / or a reverse edge processing operation.
The aircraft according to claim 51, wherein the positive edge processing operation includes:

Perform global positive edge processing on all image areas on the point cloud map to determine the image boundary of pseudo-adhesion, so as to segment each image area of pseudo-adhesion; and / or,

Perform a local positive edge processing operation on each connected image area on the point cloud map to determine a semi-adhesive image boundary, so as to divide the semi-adhesive image area among the connected image areas.
The aircraft according to claim 52, wherein the global positive edge processing operation includes:

Each semantic set image in the point cloud map is convolved with a preset calculation kernel to obtain the minimum value of the pixels in the area covered by the calculation kernel, and the minimum value is assigned to the specified pixel.
The aircraft according to claim 52, wherein the local positive edge processing operation includes:

Convolution of the semantic collection image with connected domains in the point cloud map with a preset calculation kernel to obtain the minimum value of the pixels of the area covered by the calculation kernel, and assign the minimum value to the specified pixel point.
The aircraft according to claim 51, wherein the reverse edge processing operation includes:

Each semantic set image in the point cloud map is convoluted with a preset calculation kernel to obtain the maximum value of the pixels in the area covered by the calculation kernel, and the maximum value is assigned to the specified pixel.
The aircraft according to any one of claims 53 to 55, wherein the preset calculation core is a predetermined figure with a reference point.
An aircraft, characterized in that it includes:

body;

A power system provided on the fuselage for providing flight power;

A processor for acquiring a point cloud map containing semantics; determining each image area with different semantics on the point cloud map according to the semantics on the point cloud map; Plan a flight route; control the aircraft to fly according to the flight route.
The aircraft according to claim 57, wherein when the processor obtains a point cloud map containing semantics, it is specifically used to:

Obtain the first image data captured by the camera device mounted on the aircraft;

Processing the first image data based on a semantic recognition model to obtain the semantics of each pixel in the first image data;

Generating first point cloud data containing semantics according to the position data, height data corresponding to the first image data, and the semantics of each pixel in the first image data;

A point cloud map is generated using the first point cloud data containing semantics.
The aircraft according to claim 58, wherein the processor is further used to:

Obtain the second image data captured by the camera device mounted on the aircraft;

Processing the second image data based on the semantic recognition model to obtain the semantics of each pixel in the second image data;

Generate second point cloud data containing semantics according to the position data, height data corresponding to the second image data, and the semantics of each pixel in the second image data;

Update the point cloud map using the second point cloud data.
The aircraft according to claim 59, characterized in that

The first point cloud data, the second point cloud data, and the point cloud map all contain a plurality of point data, and each point data includes position data, height data, and multiple semantics with different confidence levels;

Each point data included in the first point cloud data corresponds to each pixel in the first image data, and each point data included in the second point cloud data corresponds to the Each pixel corresponds.
The aircraft according to claim 60, wherein the confidence level is positive floating point data.
The aircraft according to claim 59, wherein when the processor uses the second point cloud data to update the point cloud map, it is specifically used to:

Compare two point data with the same position data in the second point cloud data and the point cloud map, and retain the point data with higher confidence in the two point data.
The aircraft according to claim 62, wherein when the processor compares the two point data with the same position data in the point cloud map, it is specifically used to:

Subtraction operations are performed on a plurality of semantics with different confidence levels in two point data with the same position data in the first point cloud data and the second point cloud data.
The aircraft according to claim 63, characterized in that

Two point data having the same position data in the first point cloud data and the second point cloud data correspond to two overlapping pixels in the first image data and the second image data.
The aircraft according to claim 62, wherein when the processor updates the point cloud map using the second point cloud data, the processor is specifically configured to:

Count the number of semantics of the two point data with the same position data in the first point cloud data and the second point cloud data are marked as the number of the same semantics in the history record;

The semantics with the largest number is used as the semantics of the two point data with the same position data in the first point cloud data and the second point cloud data.
The aircraft according to claim 62, wherein when the processor updates the point cloud map using the second point cloud data, the processor is specifically configured to:

According to the priorities corresponding to the semantics of the two point data with the same position data in the second point cloud data and the point cloud map, it is determined that the semantics with the highest priority are the second point cloud data and the The semantics of two point data with the same position data in a point cloud map.
The aircraft according to claim 58, characterized in that

The first image data includes a color image; or,

The first image data includes a color image and depth data corresponding to the color image; or,

The first image data includes an orthophoto; or,

The first image data includes orthophotos and depth data corresponding to the orthophotos.
The aircraft according to claim 58, wherein before processing the first image data based on a semantic recognition model, the processor is further configured to:

Acquiring a sample database, the sample database including sample image data;

Generate an initial semantic recognition model according to a preset semantic recognition algorithm;

Training and optimizing the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model;

Wherein, the sample image data includes a sample image and semantic annotation information; or, the sample image data includes a sample image, depth data corresponding to each pixel in the sample image and semantic annotation information.
The aircraft according to claim 68, wherein the processor performs training optimization on the initial semantic recognition model based on each sample image data in the sample database to obtain the semantic recognition model, specifically to:

Calling the initial semantic recognition model to identify the sample image included in the sample image data and the depth data corresponding to each pixel in the sample image to obtain a recognition result;

If the recognition result matches the semantic annotation information included in the sample image data, the model parameters of the initial semantic recognition model are optimized to obtain the semantic recognition model.
The aircraft according to claim 59, characterized in that

The point cloud map includes a plurality of image areas, the image areas are divided according to the semantics of each pixel in the point cloud map, and each image area is displayed by different display mark methods.
The aircraft according to claim 70, wherein the processor is specifically used when planning a flight route according to the semantics of each image area on the point cloud map:

Determine the obstacle area on the point cloud map according to the semantics corresponding to each image area on the point cloud map;

When planning the route, bypass the obstacle area to plan the flight route.
The aircraft according to claim 71, wherein the processor, when controlling the aircraft to fly according to the flight path, is specifically used to:

In the process of controlling the aircraft to fly according to the flight path, determine whether the semantics of the image area corresponding to the current flying position of the aircraft in the point cloud map match the semantics of the target task;

If the judgment result is yes, control the aircraft to perform the target mission;

If the judgment result is no, the aircraft is controlled to stop performing the target mission.
A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the method according to any one of claims 1 to 24.