CN111670417A

CN111670417A - Semantic map construction method, semantic map construction system, mobile platform and storage medium

Info

Publication number: CN111670417A
Application number: CN201980007922.6A
Authority: CN
Inventors: 王涛; 李思晋; 李鑫超
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd
Priority date: 2019-07-05
Filing date: 2019-07-05
Publication date: 2020-09-15
Also published as: WO2021003587A1

Abstract

A semantic map construction method, a semantic map construction system, a movable platform, a computer-readable storage medium, a movable platform, and a method for searching for a landing point. The semantic map construction method comprises the following steps: acquiring semantic segmentation information of a plurality of images (S102); and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images (S104). According to the semantic map construction method, the complete and accurate information of the entity contents of the real scene is obtained by obtaining the semantic segmentation information of the images, the spliced images are generated by splicing the images, the completeness and the reality of the scene are guaranteed, the semantic map obtained according to the semantic segmentation information of the images has high confidence, the scene understanding accuracy is improved, and the position information can be accurately obtained through the semantic map.

Description

Semantic map construction method, semantic map construction system, mobile platform and storage medium

Technical Field

The application relates to the technical field of intelligent recognition, in particular to a semantic map construction method, a semantic map construction system, a movable platform, a computer-readable storage medium, a movable platform and a method for searching for a landing point.

Background

The scene of equipment such as present aircraft, unmanned aerial vehicle is because background information is complicated, the visual angle is changeable, hardly accomplishes accurate scene understanding, and then can't guide equipment such as aircraft to fly more effectively. Therefore, a semantic map is needed to be constructed to guide the aircraft and other equipment to perform subsequent movement.

Content of application

The embodiment of the application provides a semantic map construction method, a semantic map construction system, a movable platform and a storage medium, and a more complete semantic map can be constructed.

To this end, a first aspect of the present application is to provide a semantic map construction method.

In a second aspect of the present application, a semantic map construction system is provided.

A third aspect of the present application is to provide a movable platform.

A fourth aspect of the present application is to provide a computer-readable storage medium.

A fifth aspect of the present application is to provide a movable platform.

A sixth aspect of the present application is to provide a method of searching for a landing point.

In view of this, according to a first aspect of the present application, there is provided a semantic map construction method, including: obtaining semantic segmentation information of a plurality of images; and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

According to the semantic map building method, the semantic segmentation information of the plurality of images is acquired, the plurality of images can be images of scenes with different visual angles and different background information, the semantic segmentation information of the plurality of images is beneficial to obtaining information of a plurality of entity contents of complete and accurate real scenes, the plurality of images are spliced to generate spliced images, the completeness and the authenticity of the scenes are beneficial to being ensured, the semantic map of the spliced images is acquired according to the semantic segmentation information of the plurality of images, the semantic map is enabled to better tend to the real scenes, the plurality of entity contents of the real scenes are completely and accurately reflected, the acquired semantic map has higher confidence, the scene understanding accuracy is improved, and the position information can be accurately acquired through the semantic map.

In a second aspect of the present application, a semantic map construction system is provided, where the semantic map construction system includes: a memory for storing a computer program; a processor for executing a computer program to implement: obtaining semantic segmentation information of a plurality of images; and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

The semantic map building system provided by the application has the advantages that the processor is enabled to obtain the semantic segmentation information of the plurality of images, the plurality of images can be images of scenes with different visual angles and different background information, the semantic segmentation information of the plurality of images is beneficial to obtaining the information of a plurality of entity contents of complete and accurate real scenes, the processor is used for splicing the plurality of images to generate spliced images, the completeness and the authenticity of the scenes are beneficial to being ensured, the semantic map of the spliced images is obtained according to the semantic segmentation information of the plurality of images, the semantic map can better tend to the real scenes, the plurality of entity contents of the real scenes are completely and accurately reflected, the obtained semantic map has higher confidence, the scene understanding accuracy is improved, and the position information can be accurately obtained through the semantic map.

In a third aspect of the present application, a movable platform is provided, which includes a semantic map construction system according to any one of the above technical solutions; and the acquisition device is connected with the construction system and is used for acquiring images and sending the images to the processor. The movable platform comprises the semantic map construction system in any technical scheme, so that the whole beneficial effects of the semantic map construction system in any technical scheme are achieved, and the details are not repeated herein.

In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when being executed by a processor, the computer program implements the semantic map construction method according to any one of the above technical solutions. Therefore, the beneficial effects of the semantic map construction method according to any of the above technical solutions are not repeated herein.

In a fifth aspect of the present application, a movable platform is provided, which includes a machine body, a power supply battery, a power system, an acquisition device and a controller, wherein the power supply battery is arranged on the machine body and used for supplying power to the power system, and the power system is used for providing flight power for the movable platform; the acquisition device is used for acquiring a plurality of images in the flight process of the movable platform; a controller for obtaining semantic segmentation information of a plurality of images; and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

In a sixth aspect of the present application, a method for searching for a landing point is provided, which is applicable to a movable platform, and includes the steps of:

obtaining a semantic map according to the construction method of the semantic map;

determining a drop point of the movable platform according to the semantic map;

and controlling the movable platform to land according to the landing point of the movable platform.

The application provides a movable platform includes: the mobile platform comprises a machine body, a power supply battery, a power system, an acquisition device and a controller, wherein the power supply battery is arranged on the machine body and used for supplying power to the power system; the acquisition device is used for acquiring a plurality of images in the flying process of the movable platform, acquiring semantic segmentation information of the plurality of images through the controller, the plurality of images can be images of scenes with different visual angles and different background information, which is beneficial for the controller to obtain complete and accurate information of a plurality of entity contents of the real scene through the semantic segmentation information of the plurality of images, the controller is used for splicing a plurality of images to generate spliced images, which is beneficial to ensuring the integrity and reality of scenes, the semantic map of the spliced image is obtained according to the semantic segmentation information of the plurality of images, so that the semantic map better tends to a real scene and completely and accurately reflects a plurality of entity contents of the real scene, and the acquired semantic map has higher confidence, the scene understanding accuracy is improved, and the controller can accurately acquire the position information through the semantic map.

Additional aspects and advantages of the present application will be set forth in part in the description which follows, or may be learned by practice of the present application.

Drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 shows a schematic flow diagram of a semantic map construction method of one embodiment of the present application;

FIG. 2 illustrates an image acquired by an embodiment of the present application;

FIG. 3 illustrates a semantic recognition graph of one embodiment of the present application;

FIG. 4 illustrates a rendering of an occlusion according to an embodiment of the present application;

FIG. 5 shows a schematic flow chart diagram of a semantic map construction method of another embodiment of the present application;

FIG. 6 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 7 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 8 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 9 shows an image acquired by another embodiment of the present application;

FIG. 10 illustrates a semantic recognition graph of another embodiment of the present application;

FIG. 11 shows a rendering of an occlusion of another embodiment of the present application;

FIG. 12 shows a schematic view of a drop point of an embodiment of the present application;

FIG. 13 shows a schematic view of a drop point of another embodiment of the present application;

FIG. 14 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 15 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 16 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 17 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 18 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 19 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 20 shows a schematic flow chart diagram of a semantic map construction method of yet another embodiment of the present application;

FIG. 21 shows a schematic block diagram of a semantic map construction system of an embodiment of the present application;

FIG. 22 illustrates a schematic structural view of a movable platform of one embodiment of the present application;

FIG. 23 shows a schematic structural view of a movable platform of yet another embodiment of the present application;

FIG. 24 is a schematic diagram illustrating a semantic recognition process for an image according to an embodiment of the present application;

FIG. 25 is a schematic diagram illustrating a process for obtaining landing point location information according to an embodiment of the present application.

Wherein, the correspondence between the reference numbers and the part names in fig. 21 to 23 is:

12 memory, 14 processor, 22 acquisition device, 24 body, 26 power supply battery, 28 power system, 21 controller, 20 movable platform and 10 semantic map construction system.

Detailed Description

In order that the above objects, features and advantages of the present application can be more clearly understood, the present application will be described in further detail with reference to the accompanying drawings and detailed description. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, however, the present application may be practiced in other ways than those described herein, and therefore the scope of the present application is not limited by the specific embodiments disclosed below.

A semantic map construction method, a semantic map construction system, a movable platform, a computer-readable storage medium, and a movable platform, a method of searching for a landing point according to some embodiments of the present application are described below with reference to fig. 1 to 25.

According to an embodiment of the first aspect of the present application, a semantic map construction method is provided, and fig. 1 shows a schematic flow chart of the semantic map construction method according to an embodiment of the present application. As shown in fig. 1, the semantic map construction method includes:

s102, obtaining semantic segmentation information of a plurality of images;

for example, taking a visual sensor as an example, as shown in fig. 2, for any one of the images acquired at any time, after performing semantic segmentation on fig. 2, a single-frame image semantic recognition map as shown in fig. 3 is generated, where the image semantic recognition map includes sky, ground and tree. And (3) covering the semantic recognition image on the original image to obtain a covered rendering image as shown in FIG. 4.

In a specific embodiment, a process of obtaining an occluded rendering map is shown in fig. 2 to 4, where the image shown in fig. 2 is an image obtained at any time, the image shown in fig. 2 is subjected to semantic segmentation to generate a single-frame image semantic recognition map shown in fig. 3, and the semantic recognition map is occluded on an original image, so that the occluded rendering map shown in fig. 4 can be obtained, where different background colors in fig. 3 refer to different semantic recognition results, for example, in the specific embodiment, examples may be performed by color, where sky blue represents sky blue, sky violet represents ground, sapphire blue represents a tree, lake blue represents water, yellow represents a building, and white represents other entities. It is to be understood that different semantic recognition results may be represented by different colors.

And S104, performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

Preferably, the semantic recognition map of the single-frame image is covered on the spliced image, so that the semantic map of the spliced image can be obtained.

In a specific embodiment, a plurality of images are acquired by an image acquisition device, the acquisition time of the plurality of images is continuous, and the image acquisition device includes but is not limited to: visual sensors, radar, multispectral sensors.

In an embodiment of the present application, preferably, the obtaining semantic segmentation information of the multiple images specifically includes: and performing semantic segmentation on the plurality of images through a preset convolutional neural network module to obtain semantic segmentation information of the plurality of images.

In the embodiment, the preset convolutional neural network module is used for performing semantic segmentation on the plurality of images to obtain semantic segmentation information of the plurality of images, so that information of entity contents in the images can be completely and accurately obtained, a semantic map obtained according to the semantic segmentation information of the plurality of images is favorable for completely and accurately reflecting the plurality of entity contents of a real scene, higher confidence is achieved, and the scene understanding accuracy is improved.

It is understood that semantic segmentation information for multiple images may also be obtained by other methods. In particular, Convolutional Neural Networks (CNNs) are suitable for various scene tasks, in particular for obtaining scene object semantic information and position information, and thus CNNs can be used to identify semantic information and position information of various objects of an aerial scene.

In the specific embodiment, a plurality of images are continuously acquired through an image acquisition device, the images are input into a preset convolutional neural network module to be subjected to semantic segmentation to obtain semantic separation information, and a semantic map is constructed according to a spliced image generated by splicing the semantic separation information and the images. By continuously acquiring images, a more complete and detailed semantic map can be obtained; furthermore, semantic segmentation information of a plurality of images is acquired through the convolutional neural network, so that entity scenes in the images can be acquired more accurately, and a more accurate semantic map is obtained.

FIG. 5 shows a schematic flow diagram of a semantic map construction method according to another embodiment of the present application. As shown in fig. 5, the semantic map construction method includes:

s202, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the images comprises semantic identification results of a plurality of pixel points;

s204, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence is lower than a preset threshold;

and S206, performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

In this embodiment, because the semantic segmentation information corresponding to any one of the plurality of images includes the semantic recognition results of the plurality of pixels, before the step of generating the stitched image by stitching the plurality of images, the semantic recognition results with the confidence lower than the preset threshold are deleted by obtaining the confidence of the semantic recognition result of each pixel, so that the semantic segmentation information only includes the semantic recognition results with higher confidence, that is, the semantic segmentation information can truly and completely represent the information of the entity content corresponding to the image, and further, the semantic map obtained according to the semantic segmentation information of the plurality of images has higher confidence, which can completely and accurately represent the entity content of the real scene, thereby improving the accuracy of scene understanding, and enabling the position information to be accurately obtained through the semantic map.

Further, the semantic recognition result may be entity content in a real scene corresponding to the image, and it is understood that the semantic recognition result may be multiple, corresponding to multiple entity content in the real scene. In a specific embodiment, the semantic recognition result may correspond to the sky, the ground, a tree, a building, and the like in the entity scene, and the entities in the image are semantically recognized by corresponding different pixel points according to different entities.

The semantic segmentation information comprises semantic recognition results of a plurality of pixel points and confidence degrees corresponding to the semantic recognition results, and the semantic recognition results with the confidence degrees lower than the preset threshold are deleted through comparing the confidence degrees with the preset threshold, so that the semantic segmentation information only comprises the semantic recognition results with higher confidence degrees, namely the semantic segmentation information can truly and completely represent information of entity contents corresponding to the image, and further the position information can be accurately obtained through the semantic map.

In an embodiment of the present application, a semantic map construction method is applied to a movable platform, and fig. 6 shows a schematic flow chart of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 6, the semantic map construction method includes:

s302, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the images comprises semantic identification results of a plurality of pixel points;

s304, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result with the confidence lower than a preset threshold;

s306, splicing the multiple images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the multiple images;

and S308, acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

In this embodiment, the semantic map construction method is applicable to a movable platform, which may be an airplane, an unmanned aerial vehicle, or other movable platforms meeting the requirements. The plurality of images are spliced to generate the spliced image according to the height information of the movable platform relative to the entities corresponding to the plurality of images, so that the generated spliced image has higher definition and restoration degree, the high real entities of the entities are in the spliced image, the semantic map of the spliced image has higher confidence coefficient, and the scene understanding accuracy is improved.

Preferably, the height information of the entity is measured by a monocular camera, a binocular camera or a laser on the movable platform.

Furthermore, the two-dimensional semantic map is spliced according to the recognition results of the height information and the semantic segmentation information of the movable platform relative to the entities corresponding to the multiple images, so that the semantic map of the spliced image can accurately acquire the target semantic information and the distance information in the scene, the position information acquired through the semantic map is more accurate, the movable platform can accurately move according to the target semantic information and the distance information, the moving safety and accuracy of the movable platform are improved, and the reliability of the product is improved. The target semantic information may be information of a target entity in a plurality of entity contents corresponding to the image, and the distance information may be a distance between the movable platform and the target entity. In a specific embodiment, movable platform is unmanned aerial vehicle, and target semantic information is the semantic information that the ground in the image corresponds, and distance information is the distance between unmanned aerial vehicle and the ground, through the distance between semantic information and unmanned aerial vehicle and the ground according to ground, can acquire the positional information on ground accurately, and then make unmanned aerial vehicle can land subaerial safely, accurately.

Specifically, single-frame identification can be performed on any image to obtain a semantic identification result of each pixel point, a plurality of images are continuously collected, and image splicing is performed in combination with height information of the movable platform relative to entities corresponding to the plurality of images, so that a real-time semantic map is constructed by multiple frames. It can be understood that the semantic recognition result of each pixel point can also be obtained in other manners. Specifically, when the movable platform is unmanned aerial vehicle, the height is the distance between the entity structure corresponding to the image and the unmanned aerial vehicle. Specifically, when the images are spliced, the overlapped parts of the images may be fused, for example, the confidence degrees of the recognition results of each pixel point of the overlapped parts of the images are compared, and the images of the overlapped parts are fused by retaining the images with higher confidence degrees and deleting the images with lower confidence degrees, that is, the favorable information in each image is extracted to the maximum extent, so that the fused spliced images ensure the integrity and reality of the scene, and further the semantic map has higher confidence degrees. In one embodiment of the present application, preferably, the semantic recognition result includes at least one of: buildings, sky, trees, water surfaces, floors.

In this embodiment, the semantic recognition result includes one or more of a building, a sky, a tree, a water surface, and a ground, and the plurality of types of the semantic recognition result include a plurality of entity contents in a real scene corresponding to the picture, so that the semantic result can truly and completely represent the entity contents corresponding to the picture, which is beneficial to improving the accuracy of scene understanding.

Further, the semantic recognition result may also include other content that satisfies the requirement.

FIG. 7 shows a schematic flow diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 7, the semantic map construction method includes:

s402, collecting a plurality of images according to a preset frequency;

s404, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the plurality of images comprises semantic identification results of a plurality of pixel points;

s406, obtaining the confidence coefficient of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence coefficient is lower than a preset threshold value;

s408, splicing the plurality of images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the plurality of images;

and S410, acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

In the embodiment, the images of scenes with different visual angles and different background information can be obtained by collecting the plurality of images according to the preset frequency, so that the entity contents of different visual angles, different positions and different background information in the real scene can be completely and accurately reflected through the plurality of images, the complete and accurate entity contents in the real scene can be obtained through the semantic segmentation information of the plurality of images, and the reliability and the accuracy of the semantic map are further ensured.

FIG. 8 shows a schematic flow diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 8, the semantic map construction method includes:

s502, collecting a plurality of images according to a preset frequency;

for example, as shown in fig. 9, a visual sensor is taken as an example, and after semantic segmentation is performed on fig. 9 for any one of the obtained images at any time, a single-frame image semantic recognition map as shown in fig. 10 is generated, wherein the image semantic recognition map includes sky, ground, trees, water and buildings. And (3) covering the semantic recognition image on the original image to obtain a covered rendering image as shown in FIG. 11.

In a specific embodiment, a process of obtaining an occluded rendering map is shown in fig. 9 to 11, where the image shown in fig. 9 is an image obtained at any time, a single-frame image semantic recognition map shown in fig. 10 is generated after semantic segmentation, and the semantic recognition map is occluded on an original image, so that the occluded rendering map shown in fig. 11 can be obtained, where different background colors shown in fig. 10 refer to different semantic recognition results, for example, in the specific embodiment, examples may be performed by color, such as setting sky blue to represent sky, blue purple to represent ground, sapphire blue to represent a tree, lake blue to represent water, yellow to represent a building, and white to represent other entities. It is to be understood that different semantic recognition results may be represented by different colors.

S504, semantic segmentation information of a plurality of images is obtained, wherein the semantic segmentation information corresponding to any one of the images comprises semantic identification results of a plurality of pixel points;

s506, obtaining the confidence coefficient of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence coefficient is lower than a preset threshold value;

s508, according to the height information of the movable platform relative to the entity corresponding to the multiple images, the multiple images are spliced to generate a spliced image;

s510, obtaining a semantic map of a spliced image according to semantic segmentation information of a plurality of images;

and S512, determining a drop point of the movable platform according to the semantic map.

Preferably, the semantic map of the single-frame stitched image shown in fig. 12 can be obtained by overlaying the semantic recognition map of the single-frame image on the stitched image, where a in fig. 12 represents a touchdown area displayed in the single-frame stitched image, and it is understood that a in fig. 12 can also be a specific point, and the point represents a touchdown point.

Preferably, the semantic map of the multi-frame stitched image shown in fig. 13 can be obtained by overlaying the semantic recognition map of the multi-frame image on the stitched image, where B in fig. 13 represents a touchdown area displayed in the multi-frame stitched image, and it is understood that B in fig. 13 may also be a specific point, and the point represents a touchdown point. As can be seen from comparison between landing areas in fig. 12 and fig. 13, in the embodiment shown in fig. 13, entity contents in a real scene can be completely and accurately reflected through a plurality of images, so as to construct a complete, accurate and detailed semantic map, and then the movable platform is guided to fly by using the complete and detailed semantic map, so as to improve flight controllability of the movable platform.

And S514, controlling the movable platform to land according to the landing point of the movable platform.

In the embodiment, the falling point of the movable platform is determined according to the semantic map, so that the semantic map with higher confidence and higher scene understanding accuracy can accurately acquire the position information, the falling point of the movable platform is further determined, the falling point is safe and reliable, and the movable platform is controlled to fall according to the falling point of the movable platform, so that the movable platform can safely, reliably and accurately fall to the falling point determined through the semantic map, the problem that the movable platform falls in water, on trees, buildings and the like to damage or destroy the movable platform in the related art is avoided, the service life of the movable platform is greatly prolonged, the use safety of the movable platform is improved, and the reliability of a product is improved.

FIG. 14 shows a schematic flow chart diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 14, the semantic map construction method includes:

s602, collecting a plurality of images according to a preset frequency;

s604, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the images comprises semantic identification results of a plurality of pixel points;

s606, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result with the confidence lower than a preset threshold;

s608, splicing the multiple images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the multiple images;

s610, obtaining a semantic map of the spliced image according to the semantic segmentation information of the plurality of images;

s612, determining a landing area of the movable platform according to the semantic map;

s614, selecting a landing point in the landing area according to the state information of the movable platform;

and S616, controlling the movable platform to land according to the landing point of the movable platform.

In the embodiment, the landing area of the movable platform is determined according to the semantic map, and the landing area can be an area which is obtained according to the semantic map and safely and reliably allows the movable platform to land, namely, the area which can cause the movable platform to land and has danger or destructiveness, such as water, trees, buildings and the like, is not included, so that the movable platform is prevented from being damaged or destroyed when being landed, and the service life of the movable platform is prolonged; the landing points are selected in the landing areas according to the state information of the movable platform, so that the selected landing points can be combined with the state information of the movable platform, the movable platform can be safely and reliably landed, the situation that the movable platform cannot reach the landing points smoothly or can not finish landing smoothly at the landing points due to the self state of the movable platform is avoided, the movable platform can be safely, smoothly, reliably and accurately landed at the landing points is further guaranteed, and the reliability of the movable platform is improved.

FIG. 15 shows a schematic flow chart diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 15, the semantic map construction method includes:

s702, collecting a plurality of images according to a preset frequency;

s704, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the plurality of images comprises semantic identification results of a plurality of pixel points;

s706, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result with the confidence lower than a preset threshold;

s708, splicing the multiple images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the multiple images;

s710, obtaining a semantic map of the spliced image according to the semantic segmentation information of the plurality of images;

s712, determining a landing area of the movable platform according to the semantic map;

s714, acquiring the residual electric quantity of the battery of the movable platform;

s716, selecting a landing point in the landing area according to the residual electric quantity and the semantic map;

and S718, controlling the movable platform to land according to the landing point of the movable platform.

In this embodiment, the step of selecting a landing point in the landing zone based on the state information of the movable platform is specifically defined. By acquiring the residual electric quantity of the battery of the movable platform and selecting the landing point in the landing area according to the residual electric quantity and the semantic map, the selected landing point can ensure that the movable platform can smoothly land at the landing point by utilizing the residual electric quantity, the movable equipment is prevented from being damaged or damaged due to the fact that the movable platform cannot smoothly arrive at the landing point by the residual electric quantity of the battery, the selected landing point has high accuracy, the movable platform can reliably and safely complete landing, and the service life of the movable platform is prolonged.

FIG. 16 shows a schematic flow chart diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 16, the semantic map construction method includes:

s802, collecting a plurality of images according to a preset frequency;

s804, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the images comprises semantic identification results of a plurality of pixel points;

s806, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result with the confidence lower than a preset threshold;

s808, splicing the plurality of images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the plurality of images;

s810, obtaining a semantic map of a spliced image according to the semantic segmentation information of the plurality of images;

s812, determining a touchdown area of the movable platform according to the semantic map;

s814, acquiring the residual electric quantity of the battery of the movable platform;

s816, acquiring a flight track of the movable platform, and selecting a landing point according to the flight track and the residual electric quantity;

and S818, controlling the movable platform to land according to the landing point of the movable platform.

In this embodiment, through the surplus electric quantity of the battery that obtains the flight orbit of portable platform and portable platform, select the landing point according to flight orbit and surplus electric quantity for selected landing point and flight orbit looks adaptation are favorable to portable platform to realize returning according to the flight orbit, improve the accuracy that portable platform returned to navigate, can guarantee simultaneously that portable platform utilizes surplus electric quantity to descend smoothly at the landing point, and then improve the reliability and the security that portable platform descends.

FIG. 17 shows a schematic flow diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 17, the semantic map construction method includes:

s902, collecting a plurality of images according to a preset frequency;

s904, obtaining semantic segmentation information of a plurality of images, wherein the semantic segmentation information corresponding to any one of the plurality of images comprises semantic identification results of a plurality of pixel points;

s906, obtaining the confidence coefficient of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence coefficient is lower than a preset threshold value;

s908, splicing the plurality of images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the plurality of images;

s910, obtaining a semantic map of the spliced images according to the semantic segmentation information of the plurality of images;

s912, determining a landing area of the movable platform according to the semantic map;

s914, acquiring the residual electric quantity of the battery of the movable platform;

s916, determining the remaining endurance mileage of the battery according to the remaining capacity of the battery;

s918, acquiring the flight track of the movable platform;

s920, selecting a landing point according to the remaining endurance mileage and the flight trajectory;

and S922, controlling the movable platform to land according to the landing point of the movable platform.

In the embodiment, the residual electric quantity of the battery of the movable platform and the flight track of the movable platform are respectively obtained according to the semantic map, the residual endurance mileage of the battery is determined according to the residual electric quantity of the battery, the residual electric quantity of the battery is specifically quantized into the residual endurance mileage of the battery, and the landing point is selected according to the residual endurance mileage and the flight track, so that the landing point is accurately and reasonably selected according to the quantized residual endurance mileage and the flight track, the accuracy of landing position information is improved, the movable platform can be ensured to land at the landing point safely and reliably, the return journey is completed according to the flight track to the maximum extent of the residual endurance mileage determined based on the residual electric quantity, and the return accuracy of the movable platform is improved.

FIG. 18 shows a schematic flow diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 18, the semantic map construction method includes:

s1002, collecting a plurality of images according to a preset frequency;

s1004, obtaining semantic segmentation information of the plurality of images, wherein the semantic segmentation information corresponding to any one of the plurality of images comprises semantic identification results of a plurality of pixel points;

s1006, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence is lower than a preset threshold;

s1008, splicing the multiple images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the multiple images;

s1010, obtaining a semantic map of a spliced image according to semantic segmentation information of a plurality of images;

s1012, determining a landing area of the movable platform according to the semantic map;

s1014, acquiring the residual electric quantity of the battery of the movable platform;

s1016, determining the remaining endurance mileage of the battery according to the remaining capacity of the battery;

s1018, acquiring a flight track of the movable platform;

s1020, determining the estimated return mileage of the movable platform according to the flight track and the semantic map;

s1022, taking the flying starting point of the flight track as the landing point under the condition that the estimated return mileage is less than or equal to the residual continuation of the journey mileage;

selecting a landing point in the landing area according to the residual endurance mileage and the departure point under the condition that the estimated return mileage is greater than the residual endurance mileage;

and S1024, controlling the movable platform to land according to the landing point of the movable platform.

In this embodiment, the step of selecting the landing point based on the remaining range and the flight trajectory is specifically defined. Respectively acquiring the residual electric quantity of a battery of the movable platform and acquiring the flight track of the movable platform according to the semantic map, and determines the estimated return mileage of the movable platform according to the flight path and the semantic map, namely, the estimated return mileage is the mileage of the movable platform returning to the flying point of the flying track, based on two conditions that the estimated return mileage is less than or equal to the residual endurance mileage and the estimated return mileage is greater than the residual endurance mileage, on the one hand, the situation that the estimated return mileage is less than or equal to the residual endurance mileage indicates that the movable platform can return to the starting point of the flight track by using the residual electric quantity of the battery, and then regard the flying spot of flight orbit as the landing point, further improve the accuracy nature of landing point for the movable platform can land at the flying spot safely, reliably, accurately, improved the precision that the movable platform was returned a voyage.

On the other hand, based on the condition that the estimated return mileage is greater than the remaining endurance mileage, the situation that the movable platform cannot return to the starting point of the flight track by using the remaining electric quantity of the battery is explained, the landing point is selected in the landing area, the movable platform can be guaranteed to smoothly complete landing, safe and reliable landing can be realized, the problem that the estimated return mileage is greater than the remaining endurance mileage, and the landing point is set as the starting point of the flight track, so that the movable platform cannot smoothly complete landing and is damaged or destroyed is avoided, the reliability of the movable platform is further improved, and the service life of the movable platform is prolonged.

It is understood that the takeoff point may be a starting point of the flight trajectory, a designated home point, or a point in a designated flight plan, such as another point set near the home point.

FIG. 19 shows a schematic flow diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 19, the semantic map construction method includes:

s1102, collecting a plurality of images according to a preset frequency;

s1104, semantic segmentation information of a plurality of images is acquired, wherein the semantic segmentation information corresponding to any one of the images comprises semantic identification results of a plurality of pixel points;

s1106, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence is lower than a preset threshold;

s1108, splicing the multiple images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the multiple images;

s1110, acquiring a semantic map of a spliced image according to the semantic segmentation information of the plurality of images;

s1112, determining a touchable area of the movable platform according to the semantic map;

s1114, acquiring the residual electric quantity of the battery of the movable platform;

s1116, determining the remaining endurance mileage of the battery according to the remaining capacity of the battery;

s1118, obtaining the flight track of the movable platform;

s1120, determining the estimated return mileage of the movable platform according to the flight track and the semantic map;

s1122, taking the flying starting point of the flight track as a landing point under the condition that the estimated return mileage is less than or equal to the residual endurance mileage; selecting a landing point in the landing area according to the residual endurance mileage and the departure point under the condition that the estimated return mileage is greater than the residual endurance mileage;

s1124, controlling the movable platform to land according to the landing point of the movable platform;

s1126, controlling the movable platform to carry out obstacle avoidance flight according to the semantic map; the obstacle avoidance flight comprises detour flight or climbing flight.

In the embodiment, the landing area of the movable platform is determined and the movable platform is controlled to perform barrier flight according to the semantic map, the movable platform is controlled to perform barrier avoiding flight according to the semantic map, the semantic map has higher confidence coefficient, so that the position information of the barrier of a real scene can be completely and accurately acquired, the movable platform is controlled to perform barrier flight to avoid the barrier, the improvement of the flight reliability of the movable platform is facilitated, the service life of the movable platform is prolonged, and the reliability of a product is improved.

The obstacle avoidance flight includes detour flight or climbing flight, the detour flight is that the obstacle avoidance flight bypasses an obstacle, and the climbing flight is that the obstacle avoidance flight flies upwards to pass through the obstacle.

Furthermore, the obstacle avoidance flight can be carried out in the return flight process, and the movable platform can also be used for carrying out the obstacle avoidance flight according to a semantic map, so that the flight reliability is further improved.

In an embodiment of the present application, preferably, the movable platform includes a collecting device, and the constructing method further includes: and controlling the acquisition device to acquire a plurality of images.

In the embodiment, the mode of acquiring the plurality of images in the semantic map construction method is specifically limited, and the acquisition of the plurality of images by controlling the acquisition device of the movable platform is simple to operate and easy to implement.

It can be understood that the number of the acquisition devices can be multiple, and the multiple acquisition devices can acquire images of scenes with different visual angles and different background information, so that the confidence of the semantic map can be improved. It can be understood that a plurality of collecting devices are arranged at different positions of the movable platform, so as to collect images of different flight attitudes, different visual angles and different background information of the movable platform.

In one embodiment of the present application, it is preferable that: and controlling a collecting device on one side of the movable platform facing the ground to collect a plurality of images according to the flight attitude of the movable platform.

In this embodiment, because the landing point of the movable platform is generally set on the ground, that is, the movable platform is finally the landing point on the ground, the acquisition device controlling the movable platform to face one side of the ground acquires a plurality of images according to the flight attitude of the movable platform, so as to obtain the semantic map on one side of the ground, which is beneficial to making the movable platform land on the landing point on the ground safely, reliably and accurately, and the method is strong in operability, easy to implement and suitable for popularization and application.

It can be understood that the acquisition device on one side close to the ideal landing point can acquire a plurality of images according to the ideal landing point, so that the movable platform can land on the ideal landing point safely, reliably and accurately, and the application range of the product is further expanded.

In one embodiment of the present application, preferably, the collecting device includes: radar, vision sensors, or multispectral sensors.

In this embodiment, the collecting device can be a radar, a vision sensor or a multispectral sensor, and the multiple types of the collecting device can meet the requirements of different mounting positions, different visual angle images and different background information images of the collecting device, and can meet the requirements of different costs of the movable platform, thereby being beneficial to expanding the application range of products.

It will be appreciated that the acquisition device may also be other devices that meet the requirements.

FIG. 20 shows a schematic flow diagram of a semantic map construction method according to yet another embodiment of the present application. As shown in fig. 20, the semantic map construction method includes:

s1202, receiving a takeoff instruction, and controlling an acquisition device to start so as to acquire a plurality of images;

s1204, obtaining semantic segmentation information of the plurality of images, wherein the semantic segmentation information corresponding to any one of the plurality of images comprises semantic identification results of a plurality of pixel points;

s1206, obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence is lower than a preset threshold;

s1208, splicing the multiple images to generate a spliced image according to the height information of the movable platform relative to the entity corresponding to the multiple images;

s1210, obtaining a semantic map of a spliced image according to semantic segmentation information of a plurality of images;

s1212, receiving a return flight instruction or detecting that the movable platform has a fault, and controlling the acquisition device to close;

s1214, determining a touchdown area of the movable platform according to the semantic map;

s1216, acquiring the remaining capacity of the battery of the movable platform;

s1218, determining the remaining endurance mileage of the battery according to the remaining capacity of the battery;

s1220, acquiring a flight track of the movable platform;

s1222, determining the estimated return mileage of the movable platform according to the flight track and the semantic map;

s1224, taking a flying starting point of the flight trajectory as a landing point under the condition that the estimated return mileage is less than or equal to the residual endurance mileage;

s1226, controlling the movable platform to land according to the landing point of the movable platform;

s1228, controlling the movable platform to carry out obstacle avoidance flight according to the semantic map; the obstacle avoidance flight comprises detour flight or climbing flight.

In the embodiment, the collecting device is controlled to start by receiving the takeoff instruction to collect a plurality of images, namely, the plurality of images are collected when the movable platform takes off, the semantic map is built in real time, the collecting device is controlled to close by receiving the return instruction or detecting that the movable platform breaks down, namely, when the movable platform needs to return, the collecting device is controlled to close, the collection of the images is stopped, the position information is accurately obtained according to the built semantic map, and then the landing point of the movable platform, namely the landing position information, is determined, so that the movable platform can land at the landing point safely, reliably and accurately to complete the return, the problem that the movable platform is damaged or destroyed in water, trees, buildings and the like in the related technology is avoided, the service life of the movable platform is greatly prolonged, and the use safety of the movable platform is improved, and improves the reliability of the product.

Further, on one hand, the return instruction may be a return instruction triggered by a return key selected by a user, and on the other hand, the return instruction is a return instruction sent by a controller of the movable platform when the movable platform flies to a return point of the flight trajectory. The different modes of the return flight instruction can meet the requirements of different working conditions of the movable platform, so that the application range of products is enlarged, meanwhile, the safe return flight of the movable platform can be flexibly controlled, and the reliability of the movable platform is further improved.

As shown in fig. 21, an embodiment of the second aspect of the present application proposes a semantic map construction system 10, which includes: a memory 12 for storing a computer program; a processor 14 for executing a computer program to implement: obtaining semantic segmentation information of a plurality of images; and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

The semantic map construction system 10 provided by the embodiment of the application includes a memory 12 and a processor 14, the memory 12 is used for storing a computer program, the processor 14 is enabled to obtain semantic segmentation information of a plurality of images, the plurality of images can be images of scenes with different viewing angles and different background information, information of a plurality of entity contents of a complete and accurate real scene can be obtained through the semantic segmentation information of the plurality of images, the processor 14 is enabled to perform a splicing operation on the plurality of images to generate a spliced image, which is beneficial to ensuring the completeness and reality of the scene, the semantic map of the spliced image is obtained according to the semantic segmentation information of the plurality of images, so that the semantic map better tends to the real scene, the plurality of entity contents of the real scene are completely and accurately reflected, and the obtained semantic map has higher confidence, the accuracy of scene understanding is improved, and the position information can be accurately acquired through the semantic map.

In one embodiment of the present application, the processor 14 is preferably configured to execute the semantic segmentation information for acquiring the plurality of images as follows: and performing semantic segmentation on the plurality of images through a preset convolutional neural network module to obtain semantic segmentation information of the plurality of images.

In this embodiment, the processor 14 is configured to execute the semantic segmentation information for acquiring the plurality of images specifically as follows: the processor 14 performs semantic segmentation on the plurality of images through the preset convolutional neural network module to obtain semantic segmentation information of the plurality of images, and can completely and accurately obtain information of entity contents in the images, so that a semantic map obtained according to the semantic segmentation information of the plurality of images can completely and accurately reflect the plurality of entity contents of a real scene, and the semantic map has higher confidence and improves the accuracy of scene understanding.

It will be appreciated that the processor 14 may obtain semantic segmentation information for the plurality of images by other methods. In particular, Convolutional Neural Networks (CNNs) are suitable for various scene tasks, in particular for obtaining scene object semantic information and position information, and thus CNNs can be used to identify semantic information and position information of various objects of an aerial scene.

In a specific embodiment, the processor 14 continuously acquires a plurality of images through the image acquisition device, inputs the plurality of images into a preset convolutional neural network module for semantic segmentation to obtain semantic separation information, and constructs a semantic map according to a spliced image generated by splicing the semantic separation information and the plurality of images. The processor 14 can obtain a more complete and detailed semantic map by continuously acquiring images; the further processor 14 obtains semantic segmentation information of the plurality of images through the convolutional neural network, so that an entity scene in the image can be more accurately obtained, and a more accurate semantic map can be obtained.

In an embodiment of the present application, preferably, the semantic segmentation information corresponding to any one of the multiple images includes semantic identification results of several pixel points, and the processor 14 is configured to, before the step of stitching the multiple images to generate a stitched image, further: and obtaining the confidence coefficient of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence coefficient is lower than a preset threshold value.

In this embodiment, because the semantic segmentation information corresponding to any one of the plurality of images includes semantic identification results of a plurality of pixel points, before the step of generating a stitched image by stitching the plurality of images, the processor 14 obtains the confidence level of the semantic identification result of each pixel point, and deletes the semantic identification result whose confidence level is lower than the preset threshold value, so that the semantic segmentation information only includes the semantic identification result with higher confidence level, that is, the semantic segmentation information can truly and completely represent the information of the entity content corresponding to the image, and further, the semantic map obtained according to the semantic segmentation information of the plurality of images has higher confidence level, and can completely and accurately represent the entity content of a real scene, thereby improving the accuracy of scene understanding, and enabling position information to be accurately obtained through the semantic map.

Further, the semantic recognition result may be entity content in a real scene corresponding to the image, and it is understood that the semantic recognition result may be multiple, corresponding to multiple entity content in the real scene.

In one embodiment of the present application, preferably adapted for a movable platform, wherein the processor 14 is adapted to implement: and splicing the plurality of images according to the height information of the movable platform relative to the entity corresponding to the plurality of images to generate a spliced image.

In this embodiment, the semantic map construction system 10 is suitable for a movable platform, which may be an airplane, an unmanned plane, or other movable platforms meeting the requirements. The processor 14 splices the plurality of images to generate a spliced image according to the height information of the movable platform relative to the entities corresponding to the plurality of images, so that the generated spliced image has high definition and restoration degree, the height real entities of the entities are in the spliced image, the semantic map of the spliced image has high confidence, and the scene understanding accuracy is improved.

Preferably, the height information of the entity is measured by a binocular camera on the movable platform.

Further, the processor 14 splices the two-dimensional semantic map according to the recognition result of the height information and the semantic segmentation information of the movable platform relative to the entity corresponding to the plurality of images, so that the semantic map of the spliced image can accurately acquire the target semantic information and the distance information in the scene, the position information acquired through the semantic map is more accurate, the movable platform can accurately move according to the target semantic information and the distance information, the moving safety and accuracy of the movable platform are improved, and the reliability of the product is improved. The target semantic information may be information of a target entity in a plurality of entity contents corresponding to the image, and the distance information may be a distance between the movable platform and the target entity. In a specific embodiment, movable platform is unmanned aerial vehicle, and target semantic information is the semantic information that the ground in the image corresponds, and distance information is the distance between unmanned aerial vehicle and the ground, through the distance between semantic information and unmanned aerial vehicle and the ground according to ground, can acquire the positional information on ground accurately, and then make unmanned aerial vehicle can land subaerial safely, accurately.

Specifically, the processor 14 may perform single-frame identification on any image to obtain a semantic identification result of each pixel point, continuously acquire a plurality of images, perform image splicing in combination with height information of the movable platform relative to an entity corresponding to the plurality of images, and implement multi-frame construction of a real-time semantic map. It is understood that the processor 14 may obtain the semantic recognition result of each pixel point by other manners. Specifically, when the movable platform is unmanned aerial vehicle, the height is the distance between the entity structure corresponding to the image and the unmanned aerial vehicle. Specifically, when the images are spliced, the overlapped parts of the images may be fused, for example, the confidence degrees of the recognition results of each pixel point of the overlapped parts of the images are compared, and the images of the overlapped parts are fused by retaining the images with higher confidence degrees and deleting the images with lower confidence degrees, that is, the favorable information in each image is extracted to the maximum extent, so that the fused spliced images ensure the integrity and reality of the scene, and further the semantic map has higher confidence degrees.

In one embodiment of the present application, preferably, the semantic recognition result includes at least one of: buildings, sky, trees, water surfaces, floors.

In this embodiment, the semantic recognition result includes one or more of a building, a sky, a tree, a water surface, and a ground, and the plurality of types of the semantic recognition result include a plurality of entity contents in a real scene corresponding to the picture, so that the semantic result can truly and completely represent the entity contents corresponding to the picture, which is beneficial to improving the accuracy of scene understanding. Further, the semantic recognition result may also include other content that satisfies the requirement.

In one embodiment of the present application, processor 14 is further preferably configured to implement: a plurality of images are acquired according to a preset frequency.

In this embodiment, the processor 14 may acquire the images of the scenes with different viewing angles and different background information by acquiring the plurality of images according to the preset frequency, so that the entity contents with different viewing angles, different positions and different background information in the real scene can be completely and accurately reflected by the plurality of images, which is beneficial to obtain the complete and accurate entity contents in the real scene by the semantic segmentation information of the plurality of images, thereby ensuring the reliability and accuracy of the semantic map.

In one embodiment of the present application, processor 14 is further preferably configured to implement: determining a drop point of the movable platform according to the semantic map; and controlling the movable platform to land according to the landing point of the movable platform.

In this embodiment, the processor 14 determines the landing point of the movable platform according to the semantic map, so that the semantic map with higher confidence and higher scene understanding accuracy can accurately acquire the position information, and further determine the landing point of the movable platform, and the landing point is safe and reliable, and the processor 14 controls the movable platform to land according to the landing point of the movable platform, so that the movable platform can land to the landing point determined by the semantic map safely, reliably and accurately, thereby avoiding the problem that the movable platform lands in water, on trees, buildings and the like to damage or destroy the movable platform in the related art, greatly prolonging the service life of the movable platform, improving the safety of the movable platform, and improving the reliability of the product.

In an embodiment of the present application, preferably, the processor 14 is configured to determine, according to the semantic map, the landing point of the movable platform by: determining a touchdown area of the movable platform according to the semantic map; and selecting a landing point in the landing area according to the state information of the movable platform.

In this embodiment, the processor 14 determines the landing area of the movable platform according to the semantic map, where the landing area may be an area that is obtained according to the semantic map and safely and reliably allows the movable platform to land, that is, an area that does not include the area that can cause the movable platform to land and is dangerous or destructive, such as water, trees, buildings, and the like, thereby avoiding damage or damage of the movable platform when landing and facilitating prolonging the service life of the movable platform; the processor 14 selects the landing point in the landing area according to the state information of the movable platform, so that the selected landing point can ensure that the movable platform can land safely and reliably by combining the state information of the movable platform, the situation that the movable platform cannot reach the landing point smoothly or can not finish landing smoothly at the landing point due to the self state of the movable platform is avoided, the movable platform can land at the landing point safely, smoothly, reliably and accurately, and the reliability of the movable platform is improved.

In one embodiment of the present application, processor 14 is preferably configured to implement: selecting a landing point in the landing area according to the state information of the movable platform, which specifically comprises the following steps: acquiring the residual electric quantity of a battery of the movable platform; and selecting a landing point in the landing area according to the residual electric quantity and the semantic map.

In this embodiment, the step of the processor 14 selecting a landing point in the landing zone based on the state information of the movable platform is specifically defined. The processor 14 selects the landing point in the landing area by acquiring the remaining power of the battery of the movable platform according to the remaining power and the semantic map, so that the selected landing point can ensure that the movable platform can smoothly land at the landing point by using the remaining power, the movable equipment is prevented from being damaged or damaged due to the fact that the remaining power of the battery cannot enable the movable platform to smoothly reach the landing point, the selected landing point has high accuracy, the movable platform can reliably and safely complete landing, and the service life of the movable platform is prolonged.

In one embodiment of the present application, processor 14 is further preferably configured to implement: and acquiring the flight track of the movable platform, and selecting a landing point according to the flight track and the residual electric quantity.

In this embodiment, the processor 14 selects the landing point according to the flight trajectory and the remaining power by acquiring the flight trajectory of the movable platform, so that the selected landing point is adapted to the flight trajectory, thereby facilitating the return journey of the movable platform according to the flight trajectory, improving the accuracy of the return journey of the movable platform, and simultaneously ensuring that the movable platform smoothly lands at the landing point by using the remaining power, thereby improving the reliability and safety of the landing of the movable platform.

In one embodiment of the present application, processor 14 is further preferably configured to implement: and determining the remaining endurance mileage of the battery according to the remaining electric quantity of the battery, and selecting a landing point according to the remaining endurance mileage and the flight track.

In this embodiment, the processor 14 determines the remaining endurance mileage of the battery according to the remaining capacity of the battery, and specifically quantifies the remaining capacity of the battery into the remaining endurance mileage of the battery, so that the landing point is accurately and reasonably selected according to the quantified remaining endurance mileage and the flight trajectory, which is beneficial to improving the accuracy of landing position information, and the processor can ensure that the movable platform can land at the landing point safely and reliably, and can complete return voyage according to the flight trajectory to the maximum extent based on the remaining endurance mileage determined based on the remaining capacity, thereby improving the accuracy of return voyage of the movable platform.

In an embodiment of the present application, preferably, the processor 14 is configured to implement the step of selecting the landing point according to the remaining endurance mileage and the flight trajectory, specifically: determining the estimated return mileage of the movable platform according to the flight track and the semantic map; and taking the flying starting point of the flight path as a landing point under the condition that the estimated return mileage is less than or equal to the residual endurance mileage.

In this embodiment, the step of the processor 14 selecting the drop off point based on the remaining range and the flight trajectory is specifically defined. The processor 14 determines the estimated return mileage of the movable platform according to the flight track and the semantic map, that is, the estimated return mileage is the mileage of the movable platform returning to the starting point of the flight track, and the processor 14 indicates that the movable platform can return to the starting point of the flight track by using the residual electric quantity of the battery based on the condition that the estimated return mileage is less than or equal to the residual endurance mileage, so that the starting point of the flight track is used as the landing point, the accuracy of the landing point is further improved, the movable platform can land at the starting point safely, reliably and accurately, and the return accuracy of the movable platform is improved.

In one embodiment of the present application, preferably, based on the estimated return mileage being greater than the remaining range, the processor 14 is further configured to: and selecting a landing point in the landing area according to the residual endurance mileage and the takeoff point.

In this embodiment, the processor 14 indicates that the movable platform cannot return to the start point of the flight trajectory based on the estimated return mileage being greater than the remaining endurance mileage, and selects the landing point in the landing area to ensure that the movable platform can successfully complete landing, and can safely and reliably land, thereby avoiding the problem that the estimated return mileage being greater than the remaining endurance mileage and the landing point being set as the start point of the flight trajectory, so that the movable platform cannot successfully complete landing and is damaged or destroyed, further improving the reliability of the movable platform, and prolonging the service life of the movable platform.

In one embodiment of the present application, processor 14 is further preferably configured to implement: controlling the movable platform to carry out obstacle avoidance flight according to the semantic map; the obstacle avoidance flight comprises detour flight or climbing flight.

In this embodiment, the processor 14 controls the movable platform to perform obstacle avoidance flight according to the semantic map, because the semantic map has higher confidence, the position information of the obstacle in the real scene can be completely and accurately acquired, and the processor 14 controls the movable platform to perform obstacle avoidance flight to avoid the obstacle, which is beneficial to improving the flight reliability of the movable platform, further prolonging the service life of the movable platform, and improving the reliability of the product.

In one embodiment of the present application, preferably, the movable platform comprises an acquisition device, and the processor 14 is further configured to implement: and controlling the acquisition device to acquire a plurality of images.

In one embodiment of the present application, processor 14 is further preferably configured to implement: and controlling a collecting device on one side of the movable platform facing the ground to collect a plurality of images according to the flight attitude of the movable platform.

In this embodiment, since the landing point of the movable platform is generally set on the ground, that is, the movable platform is finally the landing point on the ground, the processor 14 controls the acquisition device on the side of the movable platform facing the ground to acquire a plurality of images according to the flight attitude of the movable platform, so as to obtain the semantic map on the side of the ground, which is beneficial to safely, reliably and accurately landing the movable platform on the landing point on the ground, and the method is strong in operability, easy to implement, and suitable for popularization and application.

It can be understood that the processor 14 may also enable the acquisition device on the side close to the ideal landing point to acquire a plurality of images according to the ideal landing point, so that the movable platform can land on the ideal landing point safely, reliably and accurately, and the application range of the product is further expanded.

In one embodiment of the present application, processor 14 is further preferably configured to implement: receiving a takeoff instruction, and controlling an acquisition device to start so as to acquire a plurality of images; and receiving a return flight instruction or detecting that the movable platform breaks down, and controlling the acquisition device to be closed.

In this embodiment, the processor 14 controls the acquisition device to start by receiving the takeoff instruction to acquire a plurality of images, that is, when the movable platform takes off, the acquisition of the plurality of images is started, and a semantic map is built in real time, the processor 14 controls the acquisition device to close by receiving the return instruction or detecting that the movable platform has a fault, that is, when the movable platform needs to return, the acquisition device is controlled to close, the acquisition of the images is stopped, and the position information is accurately acquired according to the built semantic map, so as to determine the landing point of the movable platform, that is, the landing position information, so that the movable platform can land at the landing point safely, reliably and accurately to complete the return, thereby avoiding the problem that the movable platform damages or destroys the movable platform in water, trees, buildings and the like in the related art, and greatly prolonging the service life of the movable platform, the safety of the use of the movable platform is improved, and the reliability of the product is improved.

As shown in fig. 22, an embodiment of the third aspect of the present application proposes a movable platform 20, including the semantic map building system 10 of any of the above embodiments; and the acquisition device 22, the acquisition device 22 is connected with the construction system, and the acquisition device 22 is used for acquiring images and sending the images to the processor. Since the movable platform 20 includes the semantic map building system 10 according to any of the above embodiments, all the beneficial effects of the semantic map building system 10 according to any of the above embodiments are not described herein again.

In one embodiment of the present application, the acquisition device 22 comprises: radar, vision sensors, or multispectral sensors.

In this embodiment, the collecting device 22 may be a radar, a vision sensor or a multispectral sensor, and the multiple types of the collecting device 22 can meet the requirements of different installation positions of the collecting device 22, different viewing angle images and different background information images, and can meet the requirements of different costs of the movable platform 20, which is beneficial to expanding the application range of the product. It will be appreciated that the acquisition device 22 may be other devices that meet the requirements.

Embodiments of a fourth aspect of the present application propose a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the semantic map construction method of any of the above embodiments. Therefore, the method for constructing the semantic map has the beneficial effects of any technical scheme, and is not repeated herein.

In particular, computer-readable storage media may include any medium that can store or transfer information. Examples of computer readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

As shown in fig. 23, an embodiment of the fifth aspect of the present application provides a movable platform 20, which includes a body 24, a power supply battery 26 disposed on the body 24, a power system 28, an acquisition device 22, and a controller 21, wherein the power supply battery 26 is used for supplying power to the power system 28, and the power system 28 is used for providing flight power for the movable platform 20; an acquisition device 22 for acquiring a plurality of images during the flight of the movable platform 20; a controller 21 for acquiring semantic segmentation information of a plurality of images; and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

The embodiment of the present application provides a movable platform 20 including: the aircraft comprises an aircraft body 24, a power supply battery 26 arranged on the aircraft body 24, a power system 28, an acquisition device 22 and a controller 21, wherein the power supply battery 26 is used for supplying power to the power system 28, and the power system 28 is used for supplying flight power to the movable platform 20; the acquisition device 22 is used for acquiring a plurality of images during the flight of the movable platform 20, acquiring semantic segmentation information of the plurality of images through the controller 21, the plurality of images can be images of scenes with different visual angles and different background information, which is beneficial to obtaining complete and accurate information of a plurality of entity contents of real scenes through semantic segmentation information of the plurality of images, the controller 21 is used for splicing a plurality of images to generate spliced images, which is beneficial to ensuring the integrity and reality of scenes, the semantic map of the spliced image is obtained according to the semantic segmentation information of the plurality of images, so that the semantic map better tends to a real scene and completely and accurately reflects a plurality of entity contents of the real scene, and the acquired semantic map has higher confidence, the scene understanding accuracy is improved, and the position information can be accurately acquired through the semantic map.

In a specific embodiment, the image capturing device 22 captures a plurality of images, the capturing time of the plurality of images is continuous, and the image capturing device 22 includes but is not limited to: visual sensors, radar, multispectral sensors.

In one embodiment of the present application, the controller 21 is preferably specifically configured to: and performing semantic segmentation on the plurality of images through a preset convolutional neural network module to obtain semantic segmentation information of the plurality of images.

In this embodiment, the controller 21 performs semantic segmentation on the multiple images through the preset convolutional neural network module to obtain semantic segmentation information of the multiple images, so that information of entity contents in the images can be completely and accurately obtained, and a semantic map obtained according to the semantic segmentation information of the multiple images is favorable for completely and accurately reflecting the multiple entity contents of a real scene, so that the controller has a higher confidence coefficient, and improves the accuracy of scene understanding.

It is understood that the controller 21 may obtain semantic segmentation information for the plurality of images by other methods. In particular, Convolutional Neural Networks (CNNs) are suitable for various scene tasks, in particular for obtaining scene object semantic information and position information, and thus CNNs can be used to identify semantic information and position information of various objects of an aerial scene.

In a specific embodiment, the image acquisition device 22 is used for continuously acquiring a plurality of images, the controller 21 inputs the plurality of images into a preset convolutional neural network module for semantic segmentation to obtain semantic separation information, and a semantic map is constructed according to a spliced image generated by splicing the semantic separation information and the plurality of images. The controller 21 continuously collects images through the image collecting device 22, so that a more complete and detailed semantic map can be obtained; the controller 21 further obtains semantic segmentation information of the plurality of images through the convolutional neural network, so that an entity scene in the image can be more accurately obtained, and a more accurate semantic map can be obtained.

In an embodiment of the present application, preferably, the semantic segmentation information corresponding to any one of the multiple images includes semantic identification results of a plurality of pixel points, and before the step of stitching the multiple images to generate the stitched image, the controller 21 is further configured to: and obtaining the confidence coefficient of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence coefficient is lower than a preset threshold value.

In this embodiment, because the semantic segmentation information corresponding to any one of the plurality of images includes semantic identification results of a plurality of pixel points, before the step of generating a stitched image by stitching the plurality of images, the controller 21 obtains the confidence level of the semantic identification result of each pixel point, and deletes the semantic identification result whose confidence level is lower than the preset threshold value, so that the semantic segmentation information only includes the semantic identification result with higher confidence level, that is, the semantic segmentation information can truly and completely represent the information of the entity content corresponding to the image, and further, the semantic map obtained according to the semantic segmentation information of the plurality of images has higher confidence level, and can completely and accurately represent the entity content of a real scene, thereby improving the accuracy of scene understanding, and enabling position information to be accurately obtained through the semantic map.

In one embodiment of the present application, the controller 21 is preferably specifically configured to: and splicing the plurality of images according to the height information and the height information of the movable platform 20 relative to the entity corresponding to the plurality of images to generate a spliced image.

In this embodiment, the controller 21 splices the plurality of images to generate a spliced image according to the height information of the movable platform 20 relative to the entities corresponding to the plurality of images, which is beneficial to ensuring that the generated spliced image has higher definition and reduction, and the height real entities of each entity are present in the spliced image, so that the semantic map of the obtained spliced image has higher confidence, and the accuracy of scene understanding is improved.

Further, the controller 21 splices the two-dimensional semantic map according to the recognition result of the height information and the semantic segmentation information of the movable platform 20 relative to the entity corresponding to the plurality of images, so that the semantic map of the spliced image can accurately acquire the target semantic information and the distance information in the scene, the position information acquired through the semantic map is more accurate, the movable platform 20 can accurately move according to the target semantic information and the distance information, the moving safety and accuracy of the movable platform 20 are improved, and the reliability of the product is improved. The target semantic information may be information of a target entity in the plurality of entity contents corresponding to the image, and the distance information may be a distance between the movable platform 20 and the target entity. In a specific embodiment, movable platform is unmanned aerial vehicle, and target semantic information is the semantic information that the ground in the image corresponds, and distance information is the distance between unmanned aerial vehicle and the ground, through the distance between semantic information and unmanned aerial vehicle and the ground according to ground, can acquire the positional information on ground accurately, and then make unmanned aerial vehicle can land subaerial safely, accurately.

Specifically, the controller 21 may perform single-frame identification on any image to obtain a semantic identification result of each pixel point, control the acquisition device 22 to continuously acquire a plurality of images, and perform image splicing in combination with height information of the movable platform 20 relative to entities corresponding to the plurality of images, thereby implementing multi-frame construction of a real-time semantic map. It is understood that the controller 21 may also obtain the semantic recognition result of each pixel point by other manners. Specifically, when the movable platform 20 is an unmanned aerial vehicle, the height is the distance between the solid structure corresponding to the image and the unmanned aerial vehicle. Specifically, when the images are spliced, the overlapped parts of the images may be fused, for example, the confidence degrees of the recognition results of each pixel point of the overlapped parts of the images are compared, and the images of the overlapped parts are fused by retaining the images with higher confidence degrees and deleting the images with lower confidence degrees, that is, the favorable information in each image is extracted to the maximum extent, so that the fused spliced images ensure the integrity and reality of the scene, and further the semantic map has higher confidence degrees.

In an embodiment of the present application, the acquisition means 22 are preferably specifically adapted to acquire a plurality of images at a preset frequency.

In this embodiment, the acquisition device 22 acquires a plurality of images according to the preset frequency, and images of scenes with different viewing angles and different background information can be obtained, so that the controller 21 can completely and accurately reflect the entity contents of different viewing angles, different positions and different background information in the real scene through the plurality of images, which is beneficial to obtaining the complete and accurate entity contents in the real scene through the semantic segmentation information of the plurality of images, and further ensures the reliability and accuracy of the semantic map.

In one embodiment of the present application, the controller 21 is preferably specifically configured to: determining a drop point of the movable platform 20 according to the semantic map; and controlling the movable platform 20 to descend according to the descending point of the movable platform 20.

In this embodiment, the controller 21 determines the landing point of the movable platform 20 according to the semantic map, so that the semantic map with higher confidence and higher scene understanding accuracy can accurately acquire the position information, and further determine the landing point of the movable platform 20, and the landing point is safe and reliable, the controller 21 controls the power system 28 to work according to the landing point of the movable platform 20 to enable the movable platform 20 to land, so that the movable platform 20 can land to the landing point determined by the semantic map safely, reliably and accurately, thereby avoiding the problem that the movable platform 20 is damaged or destroyed when the movable platform 20 lands in water, on trees, buildings, and the like in the related art, greatly prolonging the service life of the movable platform 20, improving the safety of the use of the movable platform 20, and improving the reliability of the product.

In one embodiment of the present application, preferably, the controller 21 determines the landing point of the movable platform 20 according to the semantic map as: determining a touchdown area of the movable platform 20 based on the semantic map; a landing point is selected in the landing area based on the state information of the movable platform 20.

In this embodiment, the controller 21 determines the landing area of the movable platform 20 according to the semantic map, and the landing area may be an area that allows the movable platform 20 to land safely and reliably according to the semantic map, that is, an area that can cause the movable platform 20 to land and is dangerous or destructive, such as water, a tree, a building, and the like, is not included, so that the movable platform 20 is prevented from being damaged or destroyed when landing, and the service life of the movable platform 20 is prolonged; the controller 21 selects the landing point in the landing area according to the state information of the movable platform 20, which is beneficial to combining the state information of the movable platform 20, so that the selected landing point can ensure that the movable platform 20 can land safely and reliably, and the situation that the landing of the movable platform 20 cannot be successfully reached or successfully completed at the landing point due to the self state of the movable platform 20 is avoided, thereby further ensuring that the movable platform 20 can land at the landing point safely, smoothly, reliably and accurately and improving the reliability of the movable platform 20.

In an embodiment of the present application, preferably, the step of selecting the landing point in the landing area by the controller 21 according to the state information of the movable platform 20 is specifically: acquiring the remaining capacity of the power supply battery 26; and selecting a landing point in the landing area according to the residual electric quantity and the semantic map.

In this embodiment, the step of the controller 21 selecting a landing point in the landing area according to the state information of the movable platform 20 is specifically defined. The controller 21 selects the landing point in the landing area by acquiring the remaining power of the power supply battery 26 of the movable platform 20 according to the remaining power and the semantic map, so that the selected landing point can ensure that the movable platform 20 can smoothly land at the landing point by using the remaining power of the power supply battery 26, the movable equipment is prevented from being damaged or destroyed because the remaining power of the power supply battery 26 cannot enable the movable platform 20 to smoothly reach the landing point, the selected landing point has higher accuracy, the movable platform 20 can reliably and safely complete landing, and the service life of the movable platform 20 is prolonged.

In an embodiment of the present application, the controller 21 is further preferably configured to: and acquiring the flight track of the movable platform 20, and selecting a landing point according to the flight track and the residual electric quantity.

In this embodiment, the controller 21 selects the landing point according to the flight trajectory and the remaining power of the power supply battery 26 by obtaining the flight trajectory of the movable platform 20, so that the selected landing point is adapted to the flight trajectory, which is beneficial to the movable platform 20 to return to the home according to the flight trajectory, thereby improving the accuracy of returning the movable platform 20, and meanwhile, the movable platform 20 can be ensured to smoothly land at the landing point by using the remaining power of the power supply battery 26, thereby improving the reliability and safety of landing of the movable platform 20.

In an embodiment of the present application, the controller 21 is further preferably configured to: and determining the remaining endurance mileage of the power supply battery 26 according to the remaining capacity of the power supply battery 26, and selecting a landing point according to the remaining endurance mileage and the flight trajectory.

In this embodiment, the controller 21 determines the remaining endurance mileage of the battery according to the remaining capacity of the power supply battery 26, and specifically quantifies the remaining capacity of the power supply battery 26 into the remaining endurance mileage of the battery, so that the landing point is accurately and reasonably selected according to the quantified remaining endurance mileage and the flight trajectory, which is beneficial to improving the accuracy of landing position information, and not only can the movable platform 20 be ensured to land at the landing point safely and reliably, but also the return journey is completed according to the flight trajectory to the maximum extent based on the remaining endurance mileage determined by the remaining capacity, so that the accuracy of the return journey of the movable platform 20 is improved.

In an embodiment of the present application, preferably, the step of selecting the landing point by the controller 21 according to the remaining endurance mileage and the flight trajectory includes: determining the estimated return mileage of the movable platform 20 according to the flight track and the semantic map; and taking the flying starting point of the flight path as a landing point under the condition that the estimated return mileage is less than or equal to the residual endurance mileage.

In this embodiment, the controller 21 determines the estimated return mileage of the movable platform 20 according to the flight trajectory and the semantic map, that is, the estimated return mileage is the mileage of the departure point of the movable platform 20 returning to the flight trajectory, and based on the condition that the estimated return mileage is less than or equal to the remaining endurance mileage, it indicates that the movable platform 20 can return to the departure point of the flight trajectory by using the remaining electric quantity of the battery, and then the controller 21 takes the departure point of the flight trajectory as the landing point, so as to further improve the accuracy of the landing point, so that the movable platform 20 can land at the departure point safely, reliably and accurately, and improve the accuracy of the return journey of the movable platform 20.

In an embodiment of the present application, preferably, based on the estimated return mileage being greater than the remaining range, the controller 21 is further configured to: and selecting a landing point in the landing area according to the residual endurance mileage and the takeoff point.

In this embodiment, based on the fact that the estimated return mileage is greater than the remaining endurance mileage, it is described that the movable platform 20 cannot return to the start point of the flight trajectory using the remaining power of the battery, the controller 21 selects the landing point in the landing area to ensure that the movable platform 20 can successfully complete the landing, and can realize safe and reliable landing, thereby avoiding the problem that the estimated return mileage is greater than the remaining endurance mileage, and the landing point is set as the start point of the flight trajectory, so that the movable platform 20 cannot successfully complete the landing and is damaged or destroyed, further improving the reliability of the movable platform 20, and prolonging the service life of the movable platform 20.

In an embodiment of the present application, the controller 21 is further preferably configured to: controlling the movable platform 20 to carry out obstacle avoidance flight according to the semantic map; the obstacle avoidance flight comprises detour flight or climbing flight.

In this embodiment, the controller 21 controls the movable platform 20 to perform obstacle avoidance flight according to the semantic map, because the semantic map has higher confidence, the position information of the obstacle in the real scene can be completely and accurately acquired, and the controller 21 controls the movable platform 20 to perform obstacle avoidance flight to avoid the obstacle, which is beneficial to improving the flight reliability of the movable platform 20, further prolonging the service life of the movable platform 20, and improving the reliability of the product.

In one embodiment of the present application, preferably, the controller 21 is further configured to: the acquisition device 22 on the ground-facing side of the movable platform 20 is controlled to acquire a plurality of images according to the flying posture of the movable platform 20.

In this embodiment, since the landing point of the movable platform 20 is generally set on the ground, that is, the movable platform 20 is finally the landing point on the ground, the controller 21 controls the acquisition device 22 on the ground side of the movable platform 20 to acquire a plurality of images according to the flight attitude of the movable platform 20, so as to obtain the semantic map on the ground side, which is beneficial to safely, reliably and accurately landing the movable platform 20 on the landing point on the ground, and the operation is strong, easy to implement, and suitable for popularization and application.

It is understood that the controller 21 may also enable the acquisition device 22 on the side close to the ideal landing point to acquire a plurality of images according to the ideal landing point, so as to enable the movable platform 20 to land on the ideal landing point safely, reliably and accurately, thereby further expanding the application range of the product.

In one embodiment of the present application, preferably, the collecting device 22 comprises: radar, vision sensors, or multispectral sensors.

In one embodiment of the present application, preferably, the controller 21 is further configured to: receiving a takeoff instruction, controlling the power system 28 and the acquisition device 22 to start so as to control the movable platform 20 to fly and the acquisition device 22 to acquire a plurality of images; and receiving a return flight instruction or detecting that the movable platform 20 has a fault, and controlling the acquisition device 22 to be closed.

In this embodiment, the controller 21 controls the power system 28 to start up by receiving a takeoff instruction, the movable platform takes off, and controls the acquisition device 22 to start up to acquire a plurality of images, that is, the acquisition device 22 starts to acquire a plurality of images when the movable platform 20 takes off, the controller 21 constructs a semantic map in real time, the controller 21 controls the acquisition device 22 to close by receiving a return flight instruction or detecting that the movable platform 20 has a fault, that is, when the movable platform 20 needs to return flight, the acquisition device 22 is controlled to close, the image acquisition is stopped, the semantic map construction is stopped, and the position information is accurately acquired according to the constructed semantic map, so as to determine the landing point of the movable platform 20, so that the movable platform 20 can safely, reliably and accurately land at the landing point to complete return flight, thereby avoiding the situation that the movable platform 20 lands in water in the related art, The movable platform 20 is damaged or destroyed on trees, buildings and the like, so that the service life of the movable platform 20 is greatly prolonged, the use safety of the movable platform 20 is improved, and the reliability of products is improved.

Further, on the one hand, the return instruction may be a return instruction triggered by a return key selected by a user, and on the other hand, the return instruction is a return instruction sent by the controller 21 of the movable platform 20 when the movable platform 20 flies to a return point of the flight trajectory. The different modes of the return flight instruction can meet the requirements of different working conditions of the movable platform 20, so that the application range of the product is enlarged, meanwhile, the safe return flight of the movable platform 20 can be flexibly controlled, and the reliability of the movable platform 20 is further improved.

In a specific embodiment, the movable platform 20 of this application is unmanned aerial vehicle, and unmanned aerial vehicle among the correlation technique, the unmanned aerial vehicle scene is because complicated environment, can not find safe and reliable's landing environment usually and appear dropping in the aquatic when returning to the air, on the tree, the condition such as above the building makes unmanned aerial vehicle appear destroying, and the unmanned aerial vehicle of this application, including organism 24, locate power supply battery 26 on the organism 24, driving system 28, collection system 22 and controller 21, controller 21 is through receiving the instruction of taking off, control driving system 28 and start, unmanned aerial vehicle takes off, and control collection system 22 and start, gather a plurality of images through collection system 22 (like the radar, the vision sensor, multispectral sensor) in real time promptly when taking off, and carry out the concatenation operation with a plurality of images in real time. The multiple images may be images of scenes with different viewing angles and different background information, which are continuously collected, for example, images collected when the scene is shielded by the unmanned aerial vehicle, and the controller 21 performs semantic segmentation on the multiple images through a preset convolutional neural network module to obtain semantic segmentation information of the multiple images, wherein the semantic segmentation information corresponding to any image includes semantic identification results of a plurality of pixel points. The specific image semantic identification process may be that the preprocessed image data is sent to the network model as RGB (Red Green Blue, color mode) three-channel data, and the network output result is obtained after forward propagation in sequence, that is, after iteration of the network model. As shown in fig. 24, the format of the image data input is N × 4 × H × W, the input data is processed by a convolutional neural network formed by a plurality of "Conv + bn + Relu" operation layers, a tensor whose network output result is N × K × H × W is obtained, the tensor is processed to obtain a recognition result and a recognition confidence, and the semantic recognition result with the confidence lower than a preset threshold is deleted, so that the semantic segmentation information only includes the semantic recognition result with a higher confidence, and the semantic recognition result may include: the building, the sky, the tree, the water surface, the ground and the like, namely the semantic segmentation information can truly and completely embody the entity content corresponding to the image. Then, the controller 21 splices the two-dimensional semantic map according to the recognition result of the height information and the semantic segmentation information of the unmanned aerial vehicle relative to the entities corresponding to the multiple images, specifically, splices the multiple frames of images by combining the heights of the unmanned aerial vehicle relative to the entities corresponding to the multiple images, fuses the overlapped parts of the multiple images to obtain a spliced image, and obtains the semantic map of the spliced image according to the semantic segmentation information of the multiple images, so that the semantic map better tends to the real scene, and completely and accurately reflects the information of the content of the multiple entities of the real scene, further the obtained semantic map has higher confidence, the accuracy of scene understanding is improved, and the position information can be accurately obtained through the semantic map.

When the unmanned aerial vehicle controller 21 receives a return flight instruction or detects that the unmanned aerial vehicle has a fault, the acquisition device 22 is controlled to stop acquiring images, and according to the semantic map formed at the current moment, the semantic information and the landing point information in the unmanned aerial vehicle scene are accurately acquired by the semantic map construction method. Specifically, through the surplus electric quantity that acquires unmanned aerial vehicle's flight orbit and unmanned aerial vehicle's power supply battery 26, select the landing point according to flight orbit and surplus electric quantity, acquire accurate descending positional information, avoided among the correlation technique unmanned aerial vehicle to descend in aquatic, on the tree, the building on the equal damage or the problem of destroying unmanned aerial vehicle, prolonged unmanned aerial vehicle's life greatly, improve the security that unmanned aerial vehicle used to improve the reliability of product. Wherein, selected descending point and flight orbit looks adaptation are favorable to unmanned aerial vehicle to return according to descending positional information and navigates, improve unmanned aerial vehicle and return the accuracy of navigating, can guarantee simultaneously that unmanned aerial vehicle utilizes the smooth descending of residual capacity to descend in the descending point, and then improve the reliability and the security that unmanned aerial vehicle descended.

Because the semantic map has higher confidence, can acquire the positional information of the barrier of real scene completely, accurately, control unmanned aerial vehicle through controller 21 and keep away the barrier flight and avoid the barrier when returning the journey, be favorable to improving the reliability that unmanned aerial vehicle flies, and then extension unmanned aerial vehicle's life improves the reliability of product.

An embodiment of a sixth aspect of the present application provides a method for searching for a landing point, which is applicable to a movable platform, and includes the steps of: obtaining a semantic map according to the construction method of the semantic map provided by any one of the embodiments; determining a drop point of the movable platform according to the semantic map; and controlling the movable platform to land according to the landing point of the movable platform.

Furthermore, the falling point of the movable platform is determined according to the semantic map, so that the semantic map with higher confidence coefficient and higher scene understanding accuracy can accurately acquire position information, the falling point of the movable platform is further determined, the falling point is safe and reliable, the movable platform is controlled to fall according to the falling point of the movable platform, the movable platform can safely, reliably and accurately fall to the falling point determined through the semantic map, the problem that the movable platform falls in water, on trees, on buildings and the like to damage or destroy the movable platform in the related technology is avoided, the service life of the movable platform is greatly prolonged, the use safety of the movable platform is improved, and the reliability of products is improved.

In an embodiment of the present application, determining a landing point of the movable platform according to the semantic map specifically includes: determining a touchdown area of the movable platform according to the semantic map; and selecting a landing point in the landing area according to the state information of the movable platform.

In an embodiment of the present application, the step of selecting a landing point in the landing area according to the state information of the movable platform specifically includes: acquiring the residual electric quantity of a battery of the movable platform; and selecting a landing point in the landing area according to the residual electric quantity and the semantic map.

In one embodiment of the present application, the method of searching for a landing point further includes: and acquiring the flight track of the movable platform, and selecting a landing point according to the flight track and the residual electric quantity.

In one embodiment of the present application, the method of searching for a landing point further includes: and determining the remaining endurance mileage of the battery according to the remaining electric quantity of the battery, and selecting a landing point according to the remaining endurance mileage and the flight track.

In an embodiment of the present application, the step of selecting the landing point according to the remaining endurance mileage and the flight trajectory specifically includes: determining the estimated return mileage of the movable platform according to the flight track and the semantic map; taking the flying starting point of the flight path as a landing point under the condition that the estimated return mileage is less than or equal to the residual endurance mileage; and selecting a landing point in the landing area according to the residual endurance mileage and the departure point under the condition that the estimated return mileage is greater than the residual endurance mileage.

In a specific embodiment, a specific process of acquiring the position information of the landing point by the movable platform 20 is shown in fig. 25, the controller 21 of the movable platform 20 controls a plurality of images acquired by the acquisition device 22 in real time to be input to a Convolutional Neural Network (CNN) module, and performs semantic segmentation on the plurality of images to obtain semantic segmentation information, where the output semantic segmentation information includes a semantic recognition result and a semantic recognition confidence of a plurality of pixel points of any one of the plurality of images. Further, the semantic recognition result with the confidence coefficient lower than the preset threshold value is deleted according to the semantic confidence coefficient, so that the semantic segmentation information only comprises the semantic recognition result with the higher confidence coefficient. The method comprises the steps of splicing a plurality of images to generate a spliced image, fusing overlapped parts in the spliced image, and covering a semantic recognition image of a plurality of frames of images with semantic recognition results on the spliced image to obtain a real-time semantic map of the plurality of frames of spliced images. According to the constructed semantic map, on one hand, the semantic judgment is carried out on the target falling point through the construction method of the semantic map, so that the accurate position information of the falling point can be obtained, on the other hand, the semantic judgment and the intelligent search (such as the residual electric quantity of the battery of the movable platform 20, the flight track and the like) are carried out on the target falling point through the construction method of the semantic map, so that the accurate position information of the falling point can be obtained, the movable platform 20 can safely and reliably land, and the service life of the movable platform 20 is prolonged.

In this application, the term "plurality" means two or more unless explicitly defined otherwise. The terms "mounted," "connected," "fixed," and the like are to be construed broadly, and for example, "connected" may be a fixed connection, a removable connection, or an integral connection; "coupled" may be direct or indirect through an intermediary. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.

In the description herein, the description of the terms "one embodiment," "some embodiments," "specific embodiments," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of building a semantic map, the method comprising:

obtaining semantic segmentation information of a plurality of images;

and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

2. The method for constructing a semantic map according to claim 1, wherein the semantic segmentation information for obtaining the plurality of images is specifically:

and performing semantic segmentation on the plurality of images through a preset convolutional neural network module to obtain semantic segmentation information of the plurality of images.

3. The method for constructing the semantic map according to claim 2, wherein the semantic segmentation information corresponding to any one of the plurality of images includes semantic recognition results of a plurality of pixel points, and before the step of stitching the plurality of images to generate a stitched image, the method further includes:

and obtaining the confidence of the semantic recognition result of each pixel point, and deleting the semantic recognition result of which the confidence is lower than a preset threshold value.

4. The semantic map construction method according to claim 3, which is suitable for a movable platform, and further comprises the following steps:

and splicing the plurality of images according to the height information of the movable platform relative to the entity corresponding to the plurality of images to generate the spliced image.

5. The method for constructing the semantic map according to claim 3, wherein the semantic recognition result comprises at least one of the following: buildings, sky, trees, water surfaces, floors.

6. The method of constructing a semantic map according to claim 1, further comprising:

and acquiring the plurality of images according to a preset frequency.

7. The method of constructing a semantic map according to claim 4, further comprising:

determining a falling point of the movable platform according to the semantic map;

8. The semantic map construction method according to claim 7, wherein the determining of the drop point of the movable platform according to the semantic map specifically comprises:

determining a touchdown area of the movable platform according to the semantic map;

and selecting a landing point in the landing area according to the state information of the movable platform.

9. The method for constructing a semantic map according to claim 8, wherein the step of selecting a landing point in the landing area according to the state information of the movable platform specifically comprises:

acquiring the residual electric quantity of a battery of the movable platform;

and selecting the landing point in the landing area according to the residual electric quantity and the semantic map.

10. The method of constructing a semantic map according to claim 9, further comprising:

and acquiring the flight track of the movable platform, and selecting the landing point according to the flight track and the residual electric quantity.

11. The method of constructing a semantic map according to claim 10, further comprising:

and determining the remaining endurance mileage of the battery according to the remaining electric quantity of the battery, and selecting the landing point according to the remaining endurance mileage and the flight track.

12. The method for constructing a semantic map according to claim 11, wherein the step of selecting the landing point according to the remaining driving range and the flight trajectory specifically comprises:

determining the estimated return mileage of the movable platform according to the flight track and the semantic map;

and taking the flying starting point of the flight track as the landing point under the condition that the estimated return mileage is less than or equal to the residual endurance mileage.

13. The semantic map construction method according to claim 12, wherein based on the estimated return mileage being greater than the remaining range, the method further comprises:

and selecting the landing point in the landing area according to the residual endurance mileage and the flying point.

14. The method for constructing the semantic map according to any one of claims 7 to 13, wherein the method further comprises:

controlling the movable platform to carry out obstacle avoidance flight according to the semantic map;

wherein the obstacle avoidance flight comprises detour flight or climbing flight.

15. The semantic map construction method according to any one of claims 7 to 13, wherein the movable platform comprises an acquisition device, the construction method further comprising:

and controlling the acquisition device to acquire the plurality of images.

16. The semantic map construction method according to claim 15, further comprising: and controlling the acquisition device on one side of the movable platform facing the ground to acquire the plurality of images according to the flight attitude of the movable platform.

17. The method for constructing a semantic map according to claim 15, wherein,

the collection device comprises: radar, vision sensors, or multispectral sensors.

18. The semantic map construction method according to claim 15, further comprising: receiving a takeoff instruction, and controlling the acquisition device to start so as to acquire the plurality of images; and

and receiving a return flight instruction or detecting that the movable platform breaks down, and controlling the acquisition device to be closed.

19. A semantic map construction system, comprising:

a memory for storing a computer program;

a processor for executing the computer program to implement:

obtaining semantic segmentation information of a plurality of images;

20. The semantic map construction system according to claim 19, wherein the processor is configured to perform semantic segmentation information for obtaining a plurality of images, specifically:

21. The semantic map construction system according to claim 20, wherein the semantic segmentation information corresponding to any one of the plurality of images includes semantic recognition results of several pixel points, and the processor is configured to, before the step of stitching the plurality of images to generate a stitched image, further:

22. The semantic map construction system of claim 21, adapted for use with a mobile platform, wherein the processor is configured to:

23. The semantic map construction system of claim 21, wherein the semantic recognition results comprise at least one of: buildings, sky, trees, water surfaces, floors.

24. The semantic map construction system of claim 19, wherein the processor is further configured to implement:

and acquiring the plurality of images according to a preset frequency.

25. The semantic map construction system of claim 22, wherein the processor is further configured to implement:

26. The semantic map construction system of claim 25, wherein the processor is configured to implement:

according to the semantic map, determining the falling point of the movable platform specifically comprises the following steps:

27. The semantic map construction system of claim 26, wherein the processor is configured to implement:

the step of selecting a landing point in the landing area according to the state information of the movable platform specifically comprises the following steps:

acquiring the residual electric quantity of a battery of the movable platform;

28. The semantic map construction system of claim 27, wherein the processor is further configured to implement:

29. The semantic map construction system of claim 28, wherein the processor is further configured to implement:

30. The semantic map construction system according to claim 29, wherein the processor is configured to implement the step of selecting the landing point according to the remaining driving range and the flight trajectory, specifically:

31. The semantic map construction system of claim 30, wherein based on the estimated return range being greater than the remaining range, the processor is further configured to:

32. The semantic map construction system of any one of claims 25-31, wherein the processor is further configured to implement: controlling the movable platform to carry out obstacle avoidance flight according to the semantic map;

33. The semantic map construction system of any one of claims 25-31, wherein the movable platform comprises an acquisition device, the processor further configured to implement: and controlling the acquisition device to acquire the plurality of images.

34. The semantic map construction system of claim 33, wherein the processor is further configured to implement: and controlling the acquisition device on one side of the movable platform facing the ground to acquire the plurality of images according to the flight attitude of the movable platform.

35. The semantic map construction system of claim 33, wherein the processor is further configured to implement:

receiving a takeoff instruction, and controlling the acquisition device to start so as to acquire the plurality of images; and

36. A mobile platform comprising a semantic map building system according to any one of claims 19 to 35; and

and the acquisition device is connected with the construction system and is used for acquiring the image and sending the image to the processor.

37. The movable platform of claim 36,

38. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the method of building a semantic map according to any one of claims 1 to 18.

39. A movable platform comprises a machine body, a power supply battery arranged on the machine body, a power system, an acquisition device and a controller, wherein,

the power supply battery is used for supplying power to the power system;

the power system is used for providing flight power for the movable platform;

the acquisition device is used for acquiring a plurality of images in the flight process of the movable platform;

the controller is used for acquiring semantic segmentation information of the plurality of images; and performing splicing operation on the plurality of images to generate a spliced image, and acquiring a semantic map of the spliced image according to the semantic segmentation information of the plurality of images.

40. The movable platform of claim 39 wherein the controller is specifically configured to semantically segment the plurality of images by a preset convolutional neural network module to obtain semantic segmentation information for the plurality of images.

41. The movable platform of claim 40, wherein the semantic segmentation information corresponding to any one of the plurality of images comprises semantic recognition results of a plurality of pixel points, and before the step of stitching the plurality of images to generate the stitched image, the controller is further configured to:

42. The movable platform of claim 41 wherein the controller is specifically configured to:

and splicing the plurality of images according to the height information and the height information of the movable platform relative to the entity corresponding to the plurality of images to generate the spliced image.

43. The movable platform of claim 41, wherein the semantic recognition results include at least one of: buildings, sky, trees, water surfaces, floors.

44. The movable platform of claim 39,

the acquisition device is specifically configured to acquire the plurality of images according to a preset frequency.

45. The movable platform of claim 42, the controller being specifically configured to:

46. The movable platform of claim 45, wherein the controller determines, from the semantic map, a drop point of the movable platform as:

47. The movable platform of claim 46, wherein the controller selects a landing point in the landing area based on the state information of the movable platform by:

acquiring the residual capacity of the power supply battery;

48. The movable platform of claim 47 wherein the controller is further to:

49. The movable platform of claim 48, the controller further to:

and determining the remaining endurance mileage of the power supply battery according to the remaining electric quantity of the power supply battery, and selecting the landing point according to the remaining endurance mileage and the flight track.

50. The movable platform of claim 49, wherein the controller selects the landing point based on the remaining range and the flight trajectory by:

51. The movable platform of claim 50, wherein, based on the estimated return range being greater than the remaining range, the controller is further configured to:

52. The movable platform of any one of claims 39-51 wherein the controller is further configured to:

53. The movable platform of any one of claims 39-51 wherein the controller is further configured to: and controlling the acquisition device on one side of the movable platform facing the ground to acquire the plurality of images according to the flight attitude of the movable platform.

54. The movable platform of any one of claims 39-51,

55. The movable platform of any one of claims 39-51 wherein the controller is further configured to: receiving a takeoff instruction, and controlling the power system and the acquisition device to start so as to control the movable platform to fly and the acquisition device to acquire the plurality of images; and

56. A method for searching a landing point is suitable for a movable platform, and comprises the following steps:

the semantic map is obtained according to the construction method of the semantic map according to any one of claims 1 to 6;

57. The method for searching for a touchdown point of claim 56, wherein determining, from the semantic map, a touchdown point for the movable platform is specifically:

58. The method of searching for a touchdown point of claim 57, wherein the step of selecting a touchdown point in the touchdown area based on the state information of the movable platform comprises:

acquiring the residual electric quantity of a battery of the movable platform;

59. The method of searching for a touchdown point according to claim 58, said method further comprising:

60. The method of searching for a touchdown point according to claim 59, said method further comprising:

61. The method for searching for a touchdown point according to claim 60, wherein said step of selecting said touchdown point based on said remaining range and said flight trajectory comprises:

62. The method for searching for a touchdown point according to claim 61, wherein, based on the predicted return mileage being greater than the remaining range, further comprising: