CN112204569A

CN112204569A - Operation planning method, device and equipment combining multispectral and earth surface semantics

Info

Publication number: CN112204569A
Application number: CN201980033739.3A
Authority: CN
Inventors: 刘志鹏; 李鑫超
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd; SZ DJI Innovations Technology Co Ltd
Priority date: 2019-11-04
Filing date: 2019-11-04
Publication date: 2021-01-08
Also published as: WO2021087685A1

Abstract

Provided are an operation planning method, device and equipment combining multispectral and earth surface semantics. The method comprises the following steps: acquiring a multispectral image of a ground surface target area; according to the multispectral image, obtaining diagnostic information of the earth surface target region and a first feature map containing earth surface semantic information; and generating a working plan of the earth surface target area according to the diagnosis information and the first characteristic diagram. The method and the device realize a processing mode of planning the operation by combining the diagnosis information of the earth surface target region and the earth surface semantics, and save the labor cost.

Description

Operation planning method, device and equipment combining multispectral and earth surface semantics

Technical Field

The application relates to the technical field of images, in particular to an operation planning method, device and equipment combining multispectral and earth surface semantics.

Background

In recent years, along with the development of science and technology, unmanned aerial vehicles are more and more widely applied.

At present, multispectral images of the earth surface region can be obtained through aerial photography of an unmanned aerial vehicle, and earth surface objects can be diagnosed through the multispectral images. Since the surface area includes both objects requiring diagnosis, such as a farmland, and objects not requiring diagnosis, such as roads, and there may be problems with some objects or there may be no problems with some objects in all the objects requiring diagnosis, after obtaining the diagnosis result of the surface area, the user needs to further determine the area corresponding to the diagnosis result by referring to the color image corresponding to the multispectral image, and plan the work for the area.

In the processing mode, after the surface object is diagnosed according to the multispectral image, the region corresponding to the diagnosis result needs to be manually determined, and the operation planning for the region needs to be performed, so that the labor cost is high.

Disclosure of Invention

The embodiment of the application provides an operation planning method, device and equipment combining multispectral and earth surface semantics, which are used for solving the technical problems that the area corresponding to a diagnosis result needs to be manually determined in the prior art, and the labor cost is high for operation planning of the area.

In a first aspect, an embodiment of the present application provides an operation planning method combining multispectral and surface semantics, including:

acquiring a multispectral image of a ground surface target area;

according to the multispectral image, obtaining diagnostic information of the earth surface target region and a first feature map containing earth surface semantic information;

and generating a working plan of the earth surface target area according to the diagnosis information and the first characteristic diagram.

In a second aspect, an embodiment of the present application provides an operation planning apparatus combining multiple spectra and surface semantics, including: a processor and a memory; the memory for storing program code; the processor, invoking the program code, when executed, is configured to:

acquiring a multispectral image of a ground surface target area;

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, where the computer program includes at least one piece of code, where the at least one piece of code is executable by a computer to control the computer to perform the method according to any one of the above first aspects.

In a fourth aspect, the present application provides a computer program, which is used to implement the method of any one of the above first aspects when the computer program is executed by a computer.

In a fifth aspect, an embodiment of the present application provides an agricultural drone, where the agricultural drone executes a job in the target area based on the job plan obtained by the method of the first aspect.

In a sixth aspect, an embodiment of the present application provides an unmanned aerial vehicle, which is equipped with a multispectral image acquisition device, and the unmanned aerial vehicle includes a processor and a memory, where the memory includes an instruction, and the instruction is called to execute the following steps when the unmanned aerial vehicle flies in a target area:

acquiring a multispectral image of a ground surface target area;

In a seventh aspect, an embodiment of the present application provides a ground end device, configured to communicate with an unmanned aerial vehicle equipped with a multispectral image acquisition device, where the unmanned aerial vehicle flies in a target area and acquires a multispectral image of a ground surface target area;

the ground-side device has a memory and a processor, the memory having instructions stored therein that are invoked to perform the steps of:

The embodiment of the application provides an operation planning method, device and equipment combining multispectral and earth surface semantic, and the method, device and equipment are characterized in that diagnosis information of an earth surface target region and a first feature map containing earth surface semantic information are obtained according to a multispectral image of the earth surface target region, and the operation planning of the earth surface target region is generated according to the diagnosis information and the first feature map, so that a processing mode of performing operation planning by combining the diagnosis information of the earth surface target region and the earth surface semantic is realized, a user can directly obtain the operation planning corresponding to the diagnosis information, and compared with the mode of manually determining a region corresponding to a diagnosis result and the mode of manually determining the operation planning corresponding to the diagnosis result, the labor cost is saved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic view of an application scenario of an operation planning method combining multispectral and earth surface semantics according to an embodiment of the present application;

fig. 2 is a schematic flowchart of an operation planning method combining multi-spectrum and surface semantics according to an embodiment of the present application;

fig. 3 is a schematic flowchart of an operation planning method combining multi-spectrum and surface semantics according to another embodiment of the present application;

fig. 4 is a first schematic diagram illustrating a relationship between an operation position and an operation amount according to an embodiment of the present application;

fig. 5 is a second schematic diagram illustrating a relationship between an operation position and an operation amount according to an embodiment of the present application;

FIG. 6 is a block diagram of a method for job planning incorporating multispectral and surface semantics provided in accordance with yet another embodiment of the present application;

fig. 7 is a schematic structural diagram of an operation planning apparatus combining multi-spectrum and surface semantics according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The operation planning method combining the multispectral and the earth surface semantics provided by the embodiment of the application can be applied to a scene needing to diagnose the earth surface object, and can be specifically executed by the operation planning combining the multispectral and the earth surface semantics. An application scenario of the method may be as shown in fig. 1, specifically, the operation planning apparatus 11 combining the multispectral and the earth surface semantics may obtain a multispectral image from another apparatus/device 12, and process the multispectral image by using the operation planning method combining the multispectral and the earth surface semantics provided in the embodiment of the present application. For the specific way of the communication connection between the operation planning apparatus 11 and other apparatuses/devices 12 combining the multispectral and the earth surface semantics, the application may not be limited, and for example, the wireless communication connection may be implemented based on a bluetooth interface, or the wired communication connection may be implemented based on an RS232 interface.

It should be noted that, as for the type of the device including the operation planning apparatus 11, the embodiment of the present application may not be limited, and the device may be, for example, a desktop, an all-in-one machine, a notebook computer, a palm computer, a tablet computer, a smart phone, a remote controller with a screen, an unmanned aerial vehicle, a ground end device, and the like.

It should be noted that, in fig. 1, the job planning apparatus 11 obtains the multispectral image from another device or equipment as an example, alternatively, the job planning apparatus 11 may obtain the multispectral image in another way, and for example, the job planning apparatus 11 may obtain the multispectral image by shooting.

According to the operation planning method combining the multispectral and the earth surface semantic, the diagnosis information of the earth surface target area and the first feature map containing the earth surface semantic information are obtained according to the multispectral image of the earth surface target area, the operation planning of the earth surface target area is generated according to the diagnosis information and the first feature map, the processing mode of performing the operation planning by combining the diagnosis information of the earth surface target area and the earth surface semantic is achieved, and the labor cost is saved.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.

Fig. 2 is a schematic flowchart of an operation planning method combining multi-spectrum and surface semantic according to an embodiment of the present application, where an execution subject of the embodiment may be an operation planning device combining multi-spectrum and surface semantic, and may specifically be a processor of the operation planning device. As shown in fig. 2, the method of this embodiment may include:

step 201, obtaining a multispectral image of a surface target area.

In this step, the surface target area refers to a surface area that needs to be diagnosed by the object, and the surface target area includes the object to be diagnosed. The object to be diagnosed may be different according to the diagnosis scene. For example, when a scene is diagnosed for a field, the object to be diagnosed may be the field. When a scene is diagnosed for a tree, the object to be diagnosed may be a tree.

The multispectral image is an image obtained by repeatedly shooting the ground surface target area by a plurality of wave bands, wherein the plurality of wave bands comprise visible light wave bands and wave bands close to visible light. It should be noted that the number of the bands of the multispectral image may be several, dozens, tens, hundreds, or even thousands. When the number of the wave bands of the multispectral image is large, namely the resolution of the wave bands is high, the multispectral image can be understood as the hyperspectral image.

It should be noted that, the specific manner of obtaining the multispectral image is not limited in the present application. Optionally, the multispectral image may be obtained by shooting through a multispectral camera mounted on the unmanned aerial vehicle.

Step 202, obtaining the diagnosis information of the earth surface target region and a first feature map containing earth surface semantic information according to the multispectral image.

In this step, because the reflection of the part of the spectrum by the surface object in the surface region is different in different states, the surface object can be diagnosed according to the information of the part of the spectrum in the multispectral image. For example, the obtaining the diagnostic information of the surface target region according to the multispectral image may specifically include: and calculating the diagnosis information of the earth surface target region according to the partial spectral information of the multispectral image.

For example, the diagnostic information of the target region of the earth surface can be calculated according to the partial spectral information of the multispectral image and a preset vegetation factor calculation formula. For example, a Normalized Vegetation Index (NDVI) may be calculated according to a preset Vegetation Index calculation formula according to information of a near-infrared band and an infrared band of the multispectral image. The normalized vegetation index is one of important parameters reflecting crop growth and nutrition information, for example, according to the normalized vegetation index, diagnostic information of the nitrogen demand of crops can be obtained, and the normalized vegetation index has an important guiding effect on rational application of nitrogen fertilizer.

It should be noted that the type of the diagnostic information can be flexibly implemented according to the requirement. Illustratively, the diagnostic information may specifically include diagnostic information of nutrient content, such as diagnostic information of nitrogen content. Illustratively, the diagnostic information may specifically include diagnostic information of pest distribution.

The first feature map may be the same size as the multispectral image, e.g., 100 by 200 each. For example, the first feature map may include surface semantic information in such a manner that pixel values in the feature map may characterize surface semantics of corresponding pixels, where the surface semantics may include identifiable surface object classes.

The ground surface object types which can be identified comprise types corresponding to the objects to be diagnosed, such as farmlands. Alternatively, the identified surface object categories may include other categories besides the objects to be diagnosed, such as roads, buildings, utility poles, and the like.

For example, if the pixel value is 1 and can represent a farm field, the pixel value is 2 and can represent a road, and the pixel value is 3 and can represent a building, in the first feature map obtained according to the multispectral image, the pixel position with the pixel value of 1 is the pixel position identified as the farm field, the pixel position with the pixel value of 2 is the pixel position identified as the road, and the pixel position with the pixel value of 3 is the pixel position identified as the building.

For example, the multispectral image may be processed to identify different types of surface objects based on the features of the surface objects, thereby obtaining a first feature map.

And step 203, generating an operation plan of the earth surface target area according to the diagnosis information and the first characteristic diagram.

In this step, since the first feature map includes the surface semantic information, the region corresponding to the object to be diagnosed can be determined according to the first feature map, and the diagnosis information of the region corresponding to the object to be diagnosed can be further combined with the diagnosis information of the surface target region. Since the diagnostic information of the region corresponding to the object to be diagnosed can represent the problem state of the object to be diagnosed, such as whether nitrogen is lacking and the degree of demand of nitrogen, based on the diagnostic information of the region corresponding to the object to be diagnosed, the work plan for the diagnostic information can be determined.

It should be noted that the specific type of job planning can be flexibly implemented according to the requirement.

In the embodiment, the diagnosis information of the earth surface target region and the first feature map containing the earth surface semantic information are obtained according to the multispectral image of the earth surface target region, and the operation plan of the earth surface target region is generated according to the diagnosis information and the first feature map, so that the processing mode of performing the operation plan by combining the diagnosis information of the earth surface target region and the earth surface semantic is realized, a user can directly obtain the operation plan corresponding to the diagnosis information, and compared with the situation that the region corresponding to the diagnosis result needs to be determined manually and the operation plan for the region corresponding to the diagnosis result needs to be determined manually, the labor cost is saved.

Fig. 3 is a schematic flowchart of an operation planning method combining multispectral and surface semantic according to another embodiment of the present application, and this embodiment mainly describes an alternative implementation of generating an operation plan of the surface target area according to the diagnostic information and the first feature map on the basis of the embodiment shown in fig. 2. As shown in fig. 3, the method of this embodiment may include:

step 301, obtaining a multispectral image of a target region on the earth's surface.

It should be noted that step 301 is similar to step 201, and is not described herein again.

Step 302, obtaining the diagnosis information of the earth surface target region and a first feature map containing earth surface semantic information according to the multispectral image.

In this step, for example, the obtaining the diagnostic information of the surface target region according to the multispectral image may specifically include: processing the multispectral image using a pre-trained first neural network model to generate a second feature map characterizing diagnostic information for each location point of the surface target region.

The second feature map may be the same size as the multispectral image, e.g., 100 by 200 each. For example, the specific way in which the second feature map represents the diagnostic information of each position point of the ground surface target area may be that the pixel value in the feature map may represent the diagnostic information of the corresponding pixel, the nitrogen element may represent the nitrogen element content by taking the nitrogen element as an example, and the pest and disease damage may represent the pest and disease damage state by taking the pest and disease damage as an example.

Illustratively, the processing the multispectral image using a pre-trained first neural network model to generate a second feature map characterizing the diagnostic information at each location point of the surface target region may specifically include steps a1 and a2 as follows.

Step A1, inputting the multispectral image into the first neural network model to obtain a model output result of the first neural network model.

The model output result of the first neural network model may include confidence characteristic maps respectively output by a plurality of output channels, the plurality of output channels may correspond to a plurality of target values (value sub-ranges) determined according to the value range of the diagnostic information one to one, and the pixel value of a single confidence characteristic map is used to represent the probability that the diagnostic information of the pixel is the target value. For example, assuming that the value range of the diagnostic information is 1 to 10, the plurality of channels may respectively correspond to the values 1 to 10 one by one, and the output channel corresponding to the value i outputs the confidence feature map i, where i is equal to 1, 2, and … … 10, the pixel value in the confidence feature map i may represent the probability that the diagnostic information of the pixel is i.

And step B2, obtaining the second feature map according to the model output result of the first neural network model.

For example, a target value corresponding to a confidence feature map with the maximum pixel value at the same pixel position in a plurality of confidence feature maps corresponding to the plurality of output channels one to one may be used as the diagnostic information of the pixel position to obtain a second feature map.

It is assumed that the number of output channels of the first neural network model is 10, 10 confidence feature maps are respectively a confidence feature map 1 to a confidence feature map 10, and the confidence feature map 1 corresponds to a target value i. For example, when the pixel value of the pixel position (100 ) in the confidence feature map 1 is 70, the pixel value of the pixel position (100 ) in the confidence feature map 2 is 50, the pixel value of the pixel position (100 ) in the confidence feature map 3 is 20, and the pixel values of the pixel positions (100 ) in the confidence feature maps 4 to 10 are 20, it may be determined that the pixel position (100 ) corresponds to a value of 1, that is, the diagnostic information of the pixel position (100 ) takes a value of 1.

Because the first neural network model is capable of determining diagnostic information at a pixel level granularity, it is advantageous to improve the accuracy of obtaining diagnostic information by processing the multispectral image using the first neural network model to generate diagnostic information for the earth's surface target region.

Illustratively, the first Neural network model may be a Convolutional Neural Network (CNN) model.

For example, the obtaining a first feature map containing surface semantic information according to the multispectral image may specifically include: and processing the multispectral image by using a pre-trained second neural network model to obtain a first feature map containing surface semantic information. For example, processing the multispectral image using a pre-trained second neural network model to obtain a first feature map containing surface semantic information may specifically include the following steps B1 and B2.

And B1, inputting the multispectral image into the second neural network model to obtain a model output result of the second neural network model.

The model output result of the second neural network model may include confidence feature maps respectively output by a plurality of output channels, the plurality of output channels may correspond to a plurality of surface object categories one to one, and the pixel value of the confidence feature map of a single surface object category is used to characterize the probability that the pixel is the surface object category. For example, assuming that 3 surface object categories can be identified, which are farmland, road and building, respectively, and the output channel corresponding to farmland outputs the confidence feature map 1, the output channel corresponding to road outputs the confidence feature map 2, and the output channel corresponding to building outputs the confidence feature map 3, the pixel value in the confidence feature map 1 may represent the probability that the pixel is farmland, the pixel value in the confidence feature map 2 may represent the probability that the pixel is road, and the pixel value in the confidence feature map 3 may represent the probability that the pixel is building.

And step B2, obtaining the first feature map according to the model output result of the second neural network model.

For example, the earth surface object class corresponding to the confidence feature map with the largest pixel value at the same pixel position in the multiple confidence feature maps corresponding to the multiple output channels one to one may be used as the earth surface object class at the pixel position, so as to obtain the first feature map.

It is assumed that the number of output channels of the second neural network model is 4, the 4 confidence feature maps are respectively confidence feature map 1 to confidence feature map 4, and the confidence feature map 1 corresponds to farmland, the confidence feature map 2 corresponds to roads, the confidence feature map 3 corresponds to buildings, and the confidence feature map 4 corresponds to "others". For example, when the pixel value of the pixel position (100 ) in the confidence feature map 1 is 70, the pixel value of the pixel position (100 ) in the confidence feature map 2 is 50, the pixel value of the pixel position (100 ) in the confidence feature map 3 is 20, and the pixel value of the pixel position (100 ) in the confidence feature map 4 is 20, it can be determined that the pixel position (100 ) is the farmland. For another example, when the pixel value of the pixel position (100,80) in the confidence feature map 1 is 20, the pixel value of the pixel position (100,80) in the confidence feature map 2 is 30, the pixel value of the pixel position (100,80) in the confidence feature map 3 is 20, and the pixel value of the pixel position (100,80) in the confidence feature map 4 is 70, it may be determined that the pixel position (100,80) is other, that is, not any one of a farmland, a road, and a building.

Illustratively, the second neural network model may specifically be a convolutional neural network model.

Illustratively, the obtaining a first feature map containing surface semantic information according to the multispectral image may specifically include: and obtaining a first feature map containing surface semantic information according to partial spectral information in the multispectral image. For example, a portion of the spectral information in the multispectral image is processed using a pre-trained second neural network model to obtain a first feature map containing surface semantic information. By obtaining the first characteristic diagram according to the partial spectral information in the multispectral image, the calculation amount can be reduced, and the calculation resources can be saved. For example, the partial spectrum information used for obtaining the first feature map may include information of a red (R) spectrum, a green (G) spectrum, and a blue (B) spectrum.

Illustratively, the obtaining a first feature map containing surface semantic information according to the multispectral image may specifically include: and obtaining a first feature Map containing surface semantic information according to the multispectral image and a Depth Map (Depth Map) of the surface target region. By taking into account the height factor of the surface objects when performing the surface object class recognition, it is possible to improve the accuracy of the recognition by obtaining the depth map of the surface target area when obtaining the first feature map, for example, trees and grasslands can be distinguished from the depth map.

And step 303, determining the corresponding relation between the work position and the work amount according to the diagnosis information and the first characteristic diagram.

In this step, the operation position is used for representing a position point of an object to be diagnosed, which needs to perform agricultural machinery operation, in the ground surface target area, for example, a position point of an agricultural unmanned aerial vehicle performing pesticide spraying operation. Since the pixel points in the first feature map can correspond to the location points in the surface target area, the work location can correspond to the pixel points of the first feature map.

The diagnostic information of the surface target region may include diagnostic information of a position point corresponding to an object to be diagnosed in the surface target region, and may also include diagnostic information of a position corresponding to another surface object.

When the diagnosis information includes diagnosis information of at least one region, for example, the step 303 may specifically include the following step C1 and step C2.

And step C1, determining the correspondence between the work position and the diagnostic information according to the diagnostic information of each of the at least one region and the first characteristic map.

The at least one region may include a region where an object to be diagnosed is located, and may also include regions where other surface objects are located. Since the first feature map includes the surface semantic information, the correspondence between the position point (i.e., the work position) of the object to be diagnosed and the diagnosis information can be obtained by combining the diagnosis information of at least one region with the first feature map.

Illustratively, the step C1 may specifically include: and screening out target pixel points with the earth surface object category as an object to be diagnosed from the first characteristic diagram, establishing a corresponding relation between the operation position corresponding to the target pixel points and the diagnosis information of the region to which the target pixel points belong, and obtaining the corresponding relation between the operation position and the diagnosis information.

And step C2, obtaining the corresponding relation between the work position and the work amount according to the corresponding relation between the work position and the diagnosis information and the corresponding relation between different diagnosis information levels and the work amount and the strategy of linear transition of the work amount.

Illustratively, the step C2 may specifically include: and matching the diagnostic information corresponding to the working position with the corresponding relation between different diagnostic information levels and the working amount, establishing the corresponding relation between the working position and the working amount matched with the corresponding diagnostic information to obtain the initial corresponding relation between the working position and the working amount, and adjusting the initial corresponding relation between the working position and the working amount according to a working amount linear transition strategy to obtain the final corresponding relation between the working position and the working amount.

Specifically, the diagnostic information level to which the diagnostic information belongs can be determined according to one piece of diagnostic information, and the corresponding work amount can be further determined according to the diagnostic information level to which the diagnostic information belongs and the corresponding relationship between different diagnostic information levels and the work amount.

The value range of the diagnostic information can be classified into degree grades, namely, the grade of the diagnostic information can be classified. Taking the nitrogen element content as an example, the value range of the nitrogen element content can be divided into 4 sub-ranges respectively corresponding to 4 grades, for example, the nitrogen element quantum-containing range 1 corresponds to an extreme deficiency grade, the nitrogen element quantum-containing range 2 corresponds to a severe deficiency grade, the nitrogen element quantum-containing range 3 corresponds to a less deficiency grade, and the nitrogen element quantum-containing range 4 corresponds to a health grade. Taking the pest state as an example, the value range of the pest state can be divided into 4 sub-ranges, which correspond to 4 levels respectively, for example, the pest state sub-range 1 corresponds to an extreme severity level, the pest state sub-range 2 corresponds to a severity level, the pest state sub-range 3 corresponds to a severe level, and the pest state sub-range 4 corresponds to a health level. It should be noted that, as for the dividing manner for obtaining the diagnostic information levels, the present application is not limited, and for example, the dividing of the diagnostic information levels may be preset, or the diagnostic information levels may be divided by a user.

Corresponding amounts of work can be set for different diagnostic information levels. Taking the content of nitrogen element as an example, the workload corresponding to the extreme deficiency grade can be 100 grams of nitrogen fertilizer per square meter, the workload corresponding to the serious deficiency grade can be 70 grams of nitrogen fertilizer per square meter, and the workload corresponding to the more deficient grade can be 40 grams of nitrogen fertilizer per square meter. The workload corresponding to the health grade may be 0 grams of nitrogen fertilizer per square meter. It should be noted that the nitrogen fertilizer application amount is only an example and is not taken as a basis for agricultural production, and the specific fertilizer application amount needs to be set according to the actual needs of crops. It should be noted that, the corresponding relationship between different diagnostic information levels and the work amount is not limited in the present application, and for example, the corresponding relationship between the diagnostic information level and the work amount may be preset, or the corresponding relationship between the diagnostic information level and the work amount may be set by a user.

Through the strategy of linear transition of the workload, the workload between the adjacent working positions can be linearly transitioned, and the large change of the workload between the adjacent working positions is avoided. It is assumed that the adjacent relationship between the working positions is as shown in fig. 4, and the working position 1 to the working position 5 belong to one area, and the working position 6 to the working position 10 belong to another area. When the linear workload transition strategy is not adopted, as shown in fig. 4, the workload from the working position 1 to the working position 5 is 100, and the workload from the working position 6 to the working position 10 is 60. When the linear workload transition strategy is adopted, as shown in fig. 5, the workload in the working positions 1 to 9 is sequentially decreased by 5, and the linear workload transition is realized. It should be noted that fig. 5 is only used as an illustration of the linear transition strategy.

Because the agricultural machinery generally can not respond to the large change of the workload in time in the operation process, the corresponding relation between the operation position and the workload is obtained according to the strategy of linear transition of the workload, the large change of the workload can be avoided, and the rationality of operation planning is improved.

When the diagnostic information is the second feature map of the diagnostic information characterizing each location point of the surface target region, for example, step 303 may specifically include the following steps D1 and D2.

And D1, determining the corresponding relation between the work position and the diagnosis information according to the first characteristic diagram and the second characteristic diagram.

Since the second feature map can represent the diagnostic information of each position point of the surface target region, and the first feature map includes surface semantic information, the correspondence between the position point (i.e., the work position) of the object to be diagnosed and the diagnostic information can be obtained by combining the second feature map with the first feature map.

Illustratively, the step D1 may specifically include: and screening out target pixel points of which the earth surface object types are objects to be diagnosed from the first characteristic diagram, and establishing a corresponding relation between the operation position corresponding to the target pixel points and the diagnosis information corresponding to the target pixel points in the second characteristic diagram, thereby obtaining the corresponding relation between the operation position and the diagnosis information.

And a step D2 of obtaining the corresponding relation between the work position and the work amount according to the corresponding relation between the work position and the diagnosis information and the corresponding relation between the different diagnosis information and the work amount.

Illustratively, the step D2 may specifically include: matching the diagnosis information corresponding to the working position with the corresponding relation between different diagnosis information and the working amount, establishing the corresponding relation between the working position and the working amount matched by the corresponding diagnosis information, thereby obtaining the corresponding relation between the working position and the working amount,

the corresponding relation exists between different diagnosis information and the workload, the workload corresponding of the diagnosis information granularity is realized through the corresponding relation between the different diagnosis information and the workload, and the precision of the workload can be improved compared with the workload corresponding of the diagnosis information grade granularity. It should be noted that, the corresponding relationship between different diagnostic information and the work amount is not limited in the present application, and for example, the corresponding relationship between the diagnostic information and the work amount may be preset, or the corresponding relationship between the diagnostic information and the work amount may be set by a user.

It should be noted that, step D1 and step D2 are a method of determining the work amount by combining the first characteristic map with the diagnostic information, and alternatively, the work amount may be determined based on the diagnostic information and then combined with the first characteristic map.

And 304, generating a work plan of the earth surface target area according to the corresponding relation between the work position and the work amount.

In this step, for example, step 304 may specifically include: and marking the corresponding relation between the operation position and the operation amount in the target image to generate an operation planning map of the earth surface target area. Illustratively, the target image includes one or more of: the multispectral image comprises a completely black image, a completely white image, the multispectral image or an image corresponding to partial spectral information in the multispectral image. The full black image may be an image in which the R value, the G value, and the B value of each pixel are all 0, and the full white image may be an image in which the R value, the G value, and the B value of each pixel are all 255. The image corresponding to part of the spectral information in the multispectral image may be specifically an RGB image corresponding to the multispectral image.

The corresponding relation between the working position and the working amount of the earth surface target area is marked in the target image, so that the working planning map of the earth surface target area is generated, and the corresponding relation between the working position and the working amount of the earth surface target area is visually represented in the way of the working planning map.

Illustratively, the diagnosis information comprises diagnosis information of pest and disease damage distribution, and the operation planning map is a pesticide application planning map. The operation planning chart can help a user to accurately operate an earth surface object (such as a farmland), the problem of using a large amount of pesticides is solved, on one hand, the usage amount of the pesticides can be reduced, the production cost is reduced, and on the other hand, the pollution to the environment due to excessive use of the pesticides can be avoided.

Illustratively, the diagnostic information comprises diagnostic information of nutrient content, and the operation planning map is a fertilization planning map. Can help the user to carry out accurate operation to earth's surface object (for example farmland) through the operation planning picture, solve the problem with how much fertilizer, can reduce fertilizer use amount on the one hand, reduce manufacturing cost, on the other hand can avoid because the pollution of excessive use fertilizer to the environment.

Illustratively, step 304 may specifically include: and generating a planned operation route and operation parameters corresponding to the operation route according to the corresponding relation between the operation position and the operation amount. The automatic planning of the operation route and the operation parameters is realized, and the labor energy cost is further saved. For an agricultural unmanned aerial vehicle, the operation route may specifically be a flight route, and the operation parameters may include flight altitude, flight speed, and the like in addition to the operation amount.

Illustratively, the method of this embodiment may further include: and sending the operation plan to an agricultural unmanned aerial vehicle. By sending the operation plan to the agricultural unmanned aerial vehicle, the agricultural unmanned aerial vehicle can operate according to the generated operation plan.

Illustratively, the method of this embodiment may further include: and displaying the operation plan. By displaying the operation plan, the user can know the operation plan of the earth surface target area through the display screen of the equipment comprising the operation planning device 11, and the convenience of the user for knowing the operation plan is improved.

In the embodiment, the diagnosis information of the earth surface target region and the first feature map containing the earth surface semantic information are obtained according to the multispectral image of the earth surface target region, the corresponding relation between the operation position and the operation amount is determined according to the diagnosis information and the first feature map, and the operation plan of the earth surface target region is generated according to the corresponding relation between the operation position and the operation amount, so that the processing mode of performing the operation plan by combining the diagnosis information and the earth surface semantic of the earth surface target region is realized, a user can directly obtain the operation plan corresponding to the diagnosis information, and the labor cost is saved.

The operation planning map can also be understood as a prescription map for problems of the object to be diagnosed, namely, the operation prescription map takes the diagnosis information including the content of nutrient elements or the state of plant diseases and insect pests, and the object to be diagnosed is a crop as an example. As shown in fig. 6, in the first step, data may be acquired, for example, an unmanned aerial vehicle mounted multispectral camera may be used to map crops to obtain multispectral information of the crops, and then complete multispectral images of the entire farmland may be obtained by image stitching, and in addition, a depth map may also be obtained according to the unmanned aerial vehicle mapping. And secondly, inputting the multispectral image and the depth map into a trained second convolutional neural network model, predicting the categories of farmland, roads, ground, water surface and the like and the credibility scores thereof (equivalent to the confidence coefficient characteristic map), and fusing the multispectral image and the depth map to obtain a final farmland semantic map (equivalent to the first characteristic map), so that the problem of applying and pesticide spraying to certain areas is solved. And thirdly, inputting the multispectral image into the trained first convolution neural network model to obtain diagnosis information of plant nutrient elements or plant disease and insect pest distribution, or obtaining the diagnosis information through a vegetation factor calculation formula. And fourthly, combining the farmland semantic map with plant nutrient elements or plant diseases and insect pests to obtain a nutrient element map or a plant disease and insect pest map of a farmland area, then dividing the farmland area into grades, taking the distribution of the plant diseases and insect pests as an example, dividing the farmland area into four grades of extreme severity, lightness and health, setting corresponding pesticide dosage for each grade, and generating a final farmland fertilization/pesticide application prescription map.

The farmland fertilization/pesticide application prescription chart can also be used for dividing farmland spraying areas. In an alternative embodiment, a farmland fertilization/pesticide application prescription map is obtained according to the multiple spectra, the operation area is divided into a plurality of blocks, and the operation condition of the agricultural unmanned aerial vehicle is planned according to the block division condition. For example, the varieties and the amount of pesticides/fertilizers to be sprayed in each block in a farmland operation area are obtained according to the prescription chart, and then the agricultural unmanned aerial vehicles are planned to operate, wherein the operation can be realized by a single agricultural unmanned aerial vehicle through medicine boxes with different loads, or can be realized by a plurality of unmanned aerial vehicles through medicine boxes with different loads. During operation, a plurality of unmanned aerial vehicles can be controlled to operate in a coordinated mode, for example, the operation areas of two unmanned aerial vehicles are controlled to be overlapped, so that the spraying amount of a certain area is increased.

It should be noted that the order of the second step and the third step is not limited.

Fig. 7 is a schematic structural diagram of an operation planning apparatus combining multi-spectrum and surface semantics according to an embodiment of the present application, and as shown in fig. 7, the apparatus 700 may include: a processor 701 and a memory 702.

The memory 702 for storing program code;

the processor 701, invoking the program code, when executed, is configured to perform the following:

acquiring a multispectral image of a ground surface target area;

The operation planning apparatus combining the multispectral and the earth surface semantic provided in this embodiment may be used to implement the technical solution of the foregoing method embodiment, and the implementation principle and the technical effect thereof are similar to those of the method embodiment, and are not described herein again.

In addition, the embodiment of the application also provides an agricultural unmanned aerial vehicle, and the agricultural unmanned aerial vehicle executes operation in the target area based on the operation plan obtained by the operation plan method combining the multispectral and the earth surface semantics.

The embodiment of the application further provides an unmanned aerial vehicle, which is provided with a multispectral image acquisition device, wherein the unmanned aerial vehicle comprises a processor and a memory, the memory contains instructions, and when the unmanned aerial vehicle flies in a target area, the instructions are called to execute the following steps:

acquiring a multispectral image of a ground surface target area;

The embodiment of the application further provides ground end equipment which is used for being in communication connection with the unmanned aerial vehicle carrying the multispectral image acquisition device, wherein the unmanned aerial vehicle flies in the target area and acquires the multispectral image of the ground surface target area;

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An operation planning method combining multispectral and earth surface semantics is characterized by comprising the following steps:

acquiring a multispectral image of a ground surface target area;

2. The method of claim 1, wherein said obtaining diagnostic information for said surface target region from said multispectral image comprises:

and calculating the diagnosis information of the earth surface target region according to the partial spectral information of the multispectral image.

3. The method of claim 1, wherein said obtaining diagnostic information for said surface target region from said multispectral image comprises:

processing the multispectral image using a pre-trained first neural network model to generate a second feature map characterizing diagnostic information for each location point of the surface target region.

4. The method according to claim 1, wherein obtaining a first feature map containing surface semantic information from the multispectral image comprises:

and processing the multispectral image by using a pre-trained second neural network model to obtain a first feature map containing surface semantic information.

5. The method according to claim 1, wherein obtaining a first feature map containing surface semantic information from the multispectral image comprises:

and obtaining a first characteristic diagram containing surface semantic information according to part of spectral information in the multispectral image.

6. The method of claim 5, wherein the partial spectrum information comprises information of a red spectrum, a green spectrum, and a blue spectrum.

7. The method according to claim 1, wherein obtaining a first feature map containing surface semantic information from the multispectral image comprises:

and obtaining a first feature map containing surface semantic information according to the multispectral image and the depth map of the surface target region.

8. The method of claim 1, wherein generating the job plan for the surface target area based on the diagnostic information and the first profile comprises:

determining the corresponding relation between the operation position and the operation amount according to the diagnosis information and the first characteristic diagram;

and generating a work plan of the earth surface target area according to the corresponding relation between the work position and the work amount.

9. The method of claim 8, wherein the diagnostic information includes diagnostic information for at least one region respectively;

the determining the corresponding relation between the work position and the work amount according to the diagnosis information and the first feature diagram comprises:

determining the corresponding relation between the operation position and the diagnosis information according to the diagnosis information of the at least one region and the first characteristic diagram;

and obtaining the corresponding relation between the working position and the working amount according to the corresponding relation between the working position and the diagnosis information and the corresponding relation between different diagnosis information levels and the working amount and a strategy of linear transition of the working amount.

10. The method of claim 8, wherein the diagnostic information is a second profile characterizing the diagnostic information at each location point of the surface target area;

determining the corresponding relation between the operation position and the diagnosis information according to the second characteristic diagram and the first characteristic diagram;

and obtaining the corresponding relation between the working position and the working amount according to the corresponding relation between the working position and the diagnosis information and the corresponding relation between different diagnosis information and the working amount.

11. The method of claim 8, wherein generating a job plan for the surface target area based on the correspondence between job locations and job volumes comprises:

and marking the corresponding relation between the operation position and the operation amount in the target image to generate an operation planning map of the earth surface target area.

12. The method of claim 11, wherein the target image comprises one or more of:

the multispectral image comprises a completely black image, a completely white image, the multispectral image or an image corresponding to partial spectral information in the multispectral image.

13. The method of claim 11, wherein the diagnostic information includes diagnostic information of pest distribution and the work plan is a drug delivery plan.

14. The method of claim 11, wherein the diagnostic information includes diagnostic information of nutrient content and the work plan is a fertilization plan.

15. The method of claim 8, wherein generating a job plan for the surface target area based on the correspondence between job locations and job volumes comprises:

and generating a planned operation route and operation parameters corresponding to the operation route according to the corresponding relation between the operation position and the operation amount.

16. The method of claim 15, further comprising:

and sending the operation plan to an agricultural unmanned aerial vehicle.

17. The method of claim 1, further comprising: and displaying the operation plan.

18. An operation planning device combining multispectral and earth surface semantics, comprising: a memory and a processor;

the memory for storing program code;

the processor, invoking the program code, when executed, is configured to:

acquiring a multispectral image of a ground surface target area;

19. A computer-readable storage medium, having stored thereon a computer program comprising at least one code section executable by a computer for controlling the computer to perform the method according to any one of claims 1-17.

20. A computer program for implementing the method according to any one of claims 1-17 when the computer program is executed by a computer.

21. An agricultural drone, characterized in that it performs work on said target area based on a work plan obtained by the method of one of claims 1 to 17.

22. The agricultural drone of claim 21, wherein the performing comprises pesticide spraying based on the obtained pest distribution of the target area.

23. The agricultural drone of claim 21, wherein the performing comprises performing a fertilizing operation based on the obtained nutrient content of the target area.

24. An unmanned aerial vehicle is provided with a multispectral image acquisition device, and is characterized in that the unmanned aerial vehicle comprises a processor and a memory, the memory contains instructions, and when the unmanned aerial vehicle flies in a target area, the instructions are called to execute the following steps:

acquiring a multispectral image of a ground surface target area;

25. The drone of claim 24, wherein the obtaining diagnostic information for the surface target region from the multispectral image comprises:

26. The drone of claim 24, wherein the obtaining diagnostic information for the surface target region from the multispectral image comprises:

27. The drone of claim 24, wherein the obtaining a first feature map containing surface semantic information from the multispectral image comprises:

28. The drone of claim 24, wherein the obtaining a first feature map containing surface semantic information from the multispectral image comprises:

29. A drone according to claim 28, characterised in that the partial spectrum information includes information of the red, green and blue spectrum.

30. The drone of claim 24, wherein the obtaining a first feature map containing surface semantic information from the multispectral image comprises:

31. The drone of claim 24, wherein the generating an operational plan for the surface target area based on the diagnostic information and the first signature includes:

32. A drone according to claim 31, wherein the diagnostic information includes at least one zone-specific diagnostic information;

33. The drone of claim 32, wherein the diagnostic information is a second signature characterizing diagnostic information for each location point of the surface target area;

34. The drone of claim 31, wherein generating the job plan for the surface target area based on the correspondence between job locations and job volumes comprises:

35. A drone as claimed in claim 34, wherein the target image includes one or more of:

36. A drone as claimed in claim 34, wherein the diagnostic information includes diagnostic information of pest distribution, and the operational schedule is a drug delivery schedule.

37. The drone of claim 34, wherein the diagnostic information includes diagnostic information of nutrient content, and the operational schedule is a fertilization schedule.

38. The drone of claim 31, wherein generating the job plan for the surface target area based on the correspondence between job locations and job volumes comprises:

39. A drone according to claim 38, wherein the method further comprises:

and executing the operation in the target area according to the operation plan.

40. A drone according to claim 24, wherein the method further comprises: and displaying the operation plan.

41. A ground end device is used for being in communication connection with an unmanned aerial vehicle carrying a multispectral image acquisition device, wherein the unmanned aerial vehicle flies in a target area and acquires a multispectral image of a ground surface target area; wherein the ground-side device has a memory and a processor, the memory having instructions stored therein that are invoked to perform the steps of:

42. The surface-end device of claim 41, wherein the obtaining diagnostic information for the surface target region from the multispectral image comprises:

43. The apparatus according to claim 41 wherein said obtaining diagnostic information for said subsurface target region from said multispectral image comprises:

44. The apparatus according to claim 41, wherein said obtaining a first feature map containing surface semantic information from said multispectral image comprises:

45. The apparatus according to claim 41, wherein said obtaining a first feature map containing surface semantic information from said multispectral image comprises:

46. The apparatus of claim 41, wherein the partial spectrum information comprises information for a red spectrum, a green spectrum, and a blue spectrum.

47. The apparatus according to claim 41, wherein said obtaining a first feature map containing surface semantic information from said multispectral image comprises:

48. The apparatus of claim 41, wherein generating a job plan for the surface target area based on the diagnostic information and the first signature comprises:

49. The apparatus of claim 48, wherein the diagnostic information comprises at least one region-specific diagnostic information;

50. The apparatus of claim 49 wherein said diagnostic information is a second profile characterizing diagnostic information at each location point of said surface target area;

51. The apparatus of claim 48, wherein said generating a job plan for said surface target area based on said job location to job volume correspondence comprises:

52. The apparatus of claim 51, wherein the target image comprises one or more of:

53. The apparatus of claim 51, wherein the diagnostic information comprises diagnostic information of pest distribution and the work plan is a drug delivery plan.

54. The apparatus of claim 51, wherein the diagnostic information comprises diagnostic information of nutrient content and the work plan is a fertilization plan.

55. The apparatus of claim 48, wherein said generating a job plan for said surface target area based on said job location to job volume correspondence comprises:

56. The apparatus of claim 55, wherein the method further comprises:

and sending the operation plan to an agricultural unmanned aerial vehicle.

57. The apparatus of claim 41, wherein the method further comprises: and displaying the operation plan.