WO2022094854A1

WO2022094854A1 - Growth monitoring method for crops, and devices and storage medium

Info

Publication number: WO2022094854A1
Application number: PCT/CN2020/126731
Authority: WO
Inventors: 任创杰; 李鑫超; 李思晋
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-11-05
Filing date: 2020-11-05
Publication date: 2022-05-12

Abstract

A growth monitoring method for crops, and an unmanned aerial vehicle, a control terminal, a monitoring apparatus, and a storage medium. The growth monitoring method for crops comprises: acquiring a target image obtained by an unmanned aerial vehicle photographing a target area, wherein crops are grown in the target area; inputting the target image into a pre-trained neural network model to obtain a radius estimation map of the crops and a confidence map of a plurality of growth state categories; performing information fusion according to the radius estimation map and the confidence map of the plurality of growth state categories, so as to obtain a fusion map; and determining the growth status of the crops according to the fusion map.

Description

Crop growth monitoring method, equipment and storage medium

technical field

The present application relates to the technical field of agricultural monitoring, and in particular, to a method for monitoring the growth of crops, an unmanned aerial vehicle, a control terminal, a monitoring device and a storage medium.

Background technique

At present, to analyze the growth status of crops (such as saplings), the method of manual statistics is generally used. However, the method of manual statistics analysis is slow and has no digital records, and it is time-consuming and labor-intensive, and sometimes abnormal growth of saplings or The locations of the lack of saplings were not identified in time and therefore could not be remedied, resulting in reduced crop yields. Therefore, there is an urgent need to provide a method for monitoring the growth of crops to solve the above problems.

SUMMARY OF THE INVENTION

The method for monitoring the growth of crops, the unmanned aerial vehicle, the control terminal, the monitoring device and the storage medium provided by the embodiments of the present application can quickly determine the growth status of the crops, so that the crops can be processed in time to improve the yield of the crops.

In a first aspect, the embodiments of the present application provide a method for monitoring the growth of crops, the method comprising:

obtaining a target image obtained by photographing a target area by a drone, and the target area is planted with crops;

Inputting the target image into a pre-trained neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories;

Perform information fusion according to the radius estimation map and the confidence maps of the plurality of growth state categories to obtain a fusion map; and

The growth status of the crop is determined according to the fusion map.

In a second aspect, an embodiment of the present application further provides an unmanned aerial vehicle, and the unmanned aerial vehicle includes:

body;

PTZ, the PTZ is installed on the body;

a photographing device, wherein the photographing device is installed on the cradle, and the photographing angle of the photographing device can be adjusted by adjusting the cradle;

processor and memory;

Wherein, the memory is used to store a computer program and a pre-trained neural network model; the processor is used to execute the computer program and implement the following steps when executing the computer program:

The growth status of the crop is determined according to the fusion map.

In a third aspect, an embodiment of the present application further provides a control terminal, the control terminal is connected to the unmanned aerial vehicle in communication, and is used to control the flying of the unmanned aerial vehicle, and the unmanned aerial vehicle includes a gimbal and a The photographing device on the PTZ; the control terminal includes:

processor and memory;

The growth status of the crop is determined according to the fusion map.

In a fourth aspect, an embodiment of the present application further provides a monitoring device, the monitoring device comprising:

processor and memory;

The growth status of the crop is determined according to the fusion map.

In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor can be implemented as implemented in the present application The steps of the growth monitoring method of any one of the examples provided.

The method for monitoring the growth of crops, the drone, the control terminal, the monitoring device and the storage medium disclosed in the embodiments of the present application can accurately monitor the growth status of the crops, specifically including different growth status categories (such as normal growth, abnormal growth or lack of growth). It is convenient for crop administrators to carry out targeted treatments on crops in a timely manner, such as fertilizing, replenishing seedlings or spraying pesticides, etc., thereby increasing the yield of crops.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application;

2 is a schematic block diagram of a control system of an unmanned aerial vehicle provided by an embodiment of the present application;

3 is a schematic structural diagram of a flight control system provided by an embodiment of the present application;

4 is a schematic diagram of a scene captured by a controlled drone provided by an embodiment of the present application;

5 is a schematic flowchart of steps of a model training method provided by an embodiment of the present application;

6 is a schematic flowchart of the steps of a method for monitoring the growth of crops provided in an embodiment of the present application;

FIG. 7 is a schematic diagram illustrating a usage process of a neural network model provided by an embodiment of the present application;

8 is a schematic diagram illustrating another usage process of the neural network model provided by an embodiment of the present application;

9 is a schematic flowchart of the steps of another method for monitoring the growth of crops provided in the embodiment of the present application;

10 is a schematic flowchart of steps of a method for determining the actual position of a crop provided by an embodiment of the present application;

FIG. 11a and FIG. 11b are schematic diagrams of corresponding principles for determining the actual position of crops provided by the embodiments of the present application;

12 is a schematic block diagram of an unmanned aerial vehicle provided by an embodiment of the present application;

13 is a schematic block diagram of a control terminal provided by an embodiment of the present application;

FIG. 14 is a schematic block diagram of a monitoring apparatus provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

It should also be understood that the terminology used in the specification of the application herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It should also be further understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items .

The flowcharts shown in the figures are for illustration only, and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

At present, the analysis of the growth status of crops (such as tree seedlings, grass seedlings, vegetable seedlings, etc.) is generally carried out by manual statistical methods, that is, monitoring personnel are required to conduct statistical analysis on the spot. Such manual statistical methods are slow in analysis and have no digital records. At the same time, it is also time-consuming and labor-intensive, and sometimes the abnormal growth of saplings or the location of the lack of saplings is not detected in time, so crop managers cannot carry out targeted remedial treatment, such as fertilizing or removing abnormal growth. Pests, etc., make up for the missing seedlings, because the timely remediation cannot be carried out, the final output of the saplings will decrease.

To this end, the embodiments of the present application provide a method for monitoring the growth of crops, an unmanned aerial vehicle, a control terminal, a monitoring device, and a storage medium. By using drones and neural network technology, the monitoring efficiency and accuracy of crop growth conditions can be improved, thereby increasing crop yields.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and features in the embodiments may be combined with each other without conflict.

The method for monitoring the growth of crops provided by the embodiments of the present application specifically uses a drone to process target images obtained by photographing crops grown in a target area, so as to monitor the growth conditions of the crops. Therefore, before introducing the growth monitoring method of crops, the structure and working principle of the UAV are introduced first.

Please refer to FIG. 1 and FIG. 2 . FIG. 1 shows the structure of the drone 100 provided by the embodiment of the present application, and FIG. 2 shows the structural framework of the control system of the drone 100 provided by the embodiment of the present application. As shown in FIG. 1 and FIG. 2 , the UAV 100 may include a body 11 , a gimbal 12 , a photographing device 13 , a power system 14 , a control system 15 , and the like.

Airframe 11 may include a fuselage and a foot frame (also called landing gear). The fuselage may include a center frame and one or more arms connected to the center frame, the one or more arms extending radially from the center frame. The tripod is connected with the fuselage, and is used for supporting when the drone 100 is landed.

The pan/tilt 12 is mounted on the body 11 for mounting the photographing device 13 . The pan/tilt 12 may include three motors, that is, the pan/tilt 12 is a three-axis pan/tilt. Under the control of the control system 15 of the UAV 100, the shooting angle of the photographing device 13 can be adjusted, and the shooting angle can be understood as The angle of the direction in which the lens of the photographing device 13 faces the target to be photographed relative to the horizontal direction or the vertical direction.

In some embodiments, the pan/tilt 12 may further include a controller for controlling the movement of the pan/tilt 12 by controlling the motor of the pan/tilt, thereby adjusting the shooting angle of the shooting device 13. It should be understood that the gimbal 12 may be independent of the UAV 100 , or may be a part of the UAV 100 . It should also be understood that the motor 122 may be a DC motor or an AC motor; or, the motor 122 may be a brushless motor or a brushed motor.

The photographing device 13 can be, for example, a device for capturing images such as a camera or a video camera, and the photographing device 13 can communicate with the control system 15 and perform photographing under the control of the control system 15 . In the embodiment of the present application, the photographing device 13 is mounted on the body 11 of the drone 100 through the gimbal 12 . It can be understood that, the photographing device 13 can also be directly fixed on the body 11 of the drone 100, so that the gimbal 12 can be omitted.

In some embodiments, the photographing device 13 may be controlled to photograph the target area from a bird's-eye view to obtain the target image. Among them, the target area is planted with crops, such as a certain field plot, or an area that the user needs to monitor, such as crops such as saplings, grass seedlings, etc., so that the target image can be identified, processed and analyzed to determine the crops in the target area. growth status.

The top view angle is that the optical axis direction of the lens of the photographing device 13 is perpendicular to the target area to be photographed, or is approximately perpendicular to the target area to be photographed, and the approximately perpendicularity is, for example, 88 degrees or 92 degrees. , which is not limited here. Shooting from a bird's-eye view can improve the accuracy and efficiency of image recognition, which is conducive to monitoring the growth of crops.

In some embodiments, the photographing device 13 may include a monocular camera or a binocular camera, which is used for capturing different functions. For example, the monocular camera is used to photograph a target image of a target area, which may specifically be an RGB image, and the binocular camera may Measure the distance from the target object to the drone, and also measure the distance from the crop to the drone, that is, to get the depth map of the crop, which can be used in conjunction with the target image to improve the recognition effect of the state category and radius of the crop .

In some embodiments, the depth image can be used in combination with the target image to analyze the growth conditions of crops in the target area, so that the growth conditions of crops can be more accurately analyzed.

The power system 14 may include one or more electronic governors (referred to as ESCs for short), one or more propellers, and one or more motors corresponding to the one or more propellers, wherein the motors are connected to the electronic governors. Between the propeller and the motor, the motor and the propeller are arranged on the arm of the drone 100 . The electronic governor is used to receive the driving signal generated by the control system 15, and provide a driving current to the motor according to the driving signal, so as to control the speed of the motor and then drive the propeller to rotate, so as to provide power for the flight of the UAV 100, which makes no The human-machine 100 is capable of motion in one or more degrees of freedom. In some embodiments, the drone 100 can rotate about one or more axes of rotation.

For example, the above-mentioned rotation axis may include a roll axis (Roll), a yaw axis (Yaw), and a pitch axis (pitch). It should be understood that the motor may be a DC motor or an AC motor. In addition, the motor may be a brushless motor or a brushed motor.

Control system 15 may include a controller and a sensing system. The controller is used to control the flight of the UAV 100, for example, the flight of the UAV 100 can be controlled according to the attitude information measured by the sensing system. It should be understood that the controller can control the UAV 100 according to pre-programmed instructions, and can also control the UAV 100 by responding to one or more control instructions from the control terminal. The sensing system is used to measure the attitude information of the UAV 100, that is, the position information and state information of the UAV 100 in space, such as three-dimensional position, three-dimensional angle, three-dimensional velocity, three-dimensional acceleration, and three-dimensional angular velocity.

For example, the sensing system may include at least one of a gyroscope, an ultrasonic sensor, an electronic compass, an inertial measurement unit (Inertial Measurement Unit, IMU), a visual sensor, a global navigation satellite system, a barometer, and other sensors. For example, the global navigation satellite system may be the Global Positioning System (GPS).

The controller may include one or more processors and memory. The processor may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), and the like. The memory may be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk, or a mobile hard disk.

In the embodiment of the present application, if a drone is used to perform the method for monitoring the growth of crops, the memory of the drone needs to store a pre-trained neural network model. Among them, the input of the neural network model is the target image captured by the drone 100, and the output is the radius estimation map of the crop and the confidence map of multiple growth state categories.

In some embodiments, in order to save the storage space of the UAV, the pre-trained neural network model can be compressed, and the compressed neural network model is stored in the memory of the UAV, wherein the compression process At least including pruning and so on.

In some embodiments, the UAV 100 may further include a radar device, and the radar device may be installed on the UAV 100, specifically, may be installed on the body 11 of the UAV 100. During the flight of the UAV 100, It is used to measure the surrounding environment of the UAV 100, such as obstacles, etc., to ensure the safety of flight.

The radar device is installed on the tripod of the UAV 100, the radar device is connected to the control system 15 in communication, and the radar device transmits the collected observation data to the control system for processing by the control system 15.

It should be noted that the UAV 100 may include two or more tripods, and the radar device is mounted on one of the tripods. The radar device may also be mounted on other positions of the UAV 100, which is not specifically limited.

The radar device mainly includes a radio frequency front-end module and a signal processing module. The radio frequency front-end module may include a transmitting antenna and a receiving antenna. The transmitting antenna is used to send signals to the target, and the receiving antenna is used to receive the signal reflected by the target. The signal processing module is responsible for generating modulation Signal and process and analyze the collected intermediate frequency signal, where the target is an obstacle, such as a building, an iron tower, a tree, etc.

In the embodiment of the present application, the radar device can specifically measure the distance from the crop to the drone, and then obtain the depth map of the crop.

The methods for monitoring the growth of crops provided by the embodiments of the present application are all realized by processing the acquired target images of crops photographed by the drone 100 . Wherein, any one of the crop growth monitoring methods provided in this application can be applied to the UAV 100 , and is specifically applied to the controller of the UAV 100 .

Exemplarily, the controller of the unmanned aerial vehicle 100 is used to: obtain a target image obtained by photographing a target area by the unmanned aerial vehicle, and the target area is planted with crops; input the target image into the pre-trained neural network model, Obtain the radius estimation map of the crop and the confidence maps of multiple growth state categories; perform information fusion according to the radius estimation map and the confidence maps of the multiple growth state categories to obtain a fusion map; and according to the fusion The graph determines the growth status of the crop. Shooting target images by drones can improve the efficiency of statistical analysis. Using the neural network model to learn the target images, the radius estimation map of crops and the confidence maps of multiple growth state categories can be obtained, which can more accurately analyze the growth of crops. situation.

It should be noted that, the method for monitoring the growth of crops provided by the embodiments of the present application can be applied to other electronic devices, such as a control terminal of a flight control system, in addition to the drone.

Please refer to FIG. 3 . FIG. 3 shows a structure of a flight control system provided by an embodiment of the present application. The flight control system includes an unmanned aerial vehicle 100 and a control terminal 200 . The control terminal 200 is located at the ground end of the unmanned aerial vehicle system 100 , and can communicate with the unmanned aerial vehicle 100 in a wireless manner, so as to remotely control the unmanned aerial vehicle 100 .

Wherein, the control terminal 200 may include a remote control, a smart phone, a tablet computer or a notebook computer, and the like.

It can be understood that when the control terminal 200 executes the method for monitoring the growth of crops, the control terminal 200 should also pre-store a trained neural network model, and the input of the neural network model is the target image captured by the drone 100, and the output is Radius estimates for crops and confidence maps for multiple growth state categories.

Exemplarily, the control terminal 200 is used to: obtain a target image obtained by photographing a target area by an unmanned aerial vehicle, and the target area is planted with crops; input the target image into a pre-trained neural network model to obtain the The radius estimation map of the crop and the confidence map of multiple growth state categories; perform information fusion according to the radius estimate map and the confidence map of the multiple growth state categories to obtain a fusion map; and determine the Describe the growth status of crops. Shooting target images by drones can improve the efficiency of statistical analysis. Using the neural network model to learn the target images, the radius estimation map of crops and the confidence maps of multiple growth state categories can be obtained, which can more accurately analyze the growth of crops. situation.

In practical applications, as shown in FIG. 4 , the user can control the drone 100 to fly to a target area through the control terminal 200 , and the target area is planted with crops, and the target area is photographed by the photographing device of the drone 100 to obtain the target area. image, the drone 100 sends the captured target image to the control terminal 200, and the control terminal 200 inputs the obtained target image into the neural network model for learning, and obtains the radius estimation map of the crop and the confidence map of multiple growth state categories ; and perform information fusion according to the radius estimation map and the confidence maps of the plurality of growth state categories to obtain a fusion map; and determine the growth state of the crops according to the fusion map. Compared with manual statistics, the growth monitoring method can improve the monitoring efficiency, and at the same time can monitor the growth status of multiple growth status categories of crops, so that the user can perform targeted remedial treatment, thereby improving the yield of crops.

The method for monitoring the growth of crops provided by the embodiments of the present application requires the use of a neural network model in addition to the use of unmanned aerial vehicles. Regardless of whether the method for monitoring the growth of crops is applied to an unmanned aerial vehicle or a control terminal, these devices (unmanned aerial vehicle) and control terminal, etc.) need to store a pre-trained neural network model.

Therefore, in order to facilitate understanding, before the detailed description of the method for monitoring the growth of crops provided by the embodiments of the present application, the training method of a neural network model provided by the embodiments of the present application is first introduced. The training can also be performed in a model training device, where the model training device is, for example, a computer device including a GPU, so as to improve the training efficiency of the model.

Please refer to FIG. 5. FIG. 5 shows a training method of a neural network model provided by an embodiment of the present application. The neural network model obtained by training can output a radius estimation map of crops and a confidence map of multiple growth state categories.

As shown in FIG. 5 , the training method of the model includes steps S101 to S103.

S101. Obtain sample data, where the sample data includes multiple images of the same crop, and the images are marked with multiple growth state categories of the crop and a radius value of the crop;

S102, dividing the sample data into training data and test data;

S103. Based on the determined neural network, use the training data to perform model training, and use the test data to perform model testing, and obtain a neural network model when the trained model converges as the pre-trained neural network model.

Before model training, it is necessary to prepare sample data, and determine the neural network and its network architecture to be used. In the embodiment of this application, the neural network used is a fully convolutional upgrade network. Of course, other convolutional neural networks can also be used. Network, the network architecture adopts GoogLeNet developed by Google, of course, other network architectures can also be used, such as AlexNet or ResNet and other structures.

When preparing sample data, first determine the crops that need to be monitored, such as saplings, collect a certain number of images including the crops, and process these images, which specifically includes cropping and labeling. Cropping refers to cropping all images. For images with the same pixel, labeling refers to labeling the coordinates of the radius of crops in multiple images and labeling the growth state categories of crops, in order to expect the trained neural network model to output the estimated radius of crops and multiple growth state categories. confidence map.

In some embodiments, in order to make the trained model output a better effect, the labeling specifically includes generating a circle with Gaussian distribution on the corresponding confidence feature map and recording the normalized crop radius on the corresponding radius feature map, Generate Gaussian circles and normalized crop radii corresponding to all labeled coordinates, so that the model can output a crop radius estimate map and a confidence map of growth state categories, where the radius estimate map includes normalized crop radius, growth state category The confidence map for includes a Gaussian circle.

Among them, the growth state category includes at least one or more of the normal growth category, the abnormal growth category, and the lacking seedling growth category, wherein the abnormal growth category is a category that is different from the normal growth and lacking seedlings, such as pests or dwarf growth, etc. .

The processed images are divided into two groups, one as training data and the other as test data. Specifically, it can be grouped according to a certain ratio. For example, in the case of 9:1 or 8:2, the processed images are divided into two groups.

Input the training data into the input layer of the determined neural network, train the model, and test the trained model by using the test data. When the output result of the trained model is similar to the target result, that is, when the trained model converges , a neural network model is obtained as the pre-trained neural network model. The neural network model can learn target images including crops, and can output a radius estimation map of crops and a confidence map of multiple growth state categories.

In some embodiments, the sample data can be added to the depth map of the crops, and the model can be trained, so that the obtained neural network model takes into account the factors of the depth map of the crops, so that the trained neural network model can output more accurate crops. Radius estimates and confidence maps for multiple growth state categories.

In some embodiments, in order to further improve the monitoring accuracy of the growth status of crops, the crops can be classified to obtain the types of crops, and a neural network model can be trained for each type, such as training for sapling types, grass seedling types, vegetable types, etc. The corresponding neural network model. Of course, it is also possible to train a neural network model for each crop, and establish a corresponding relationship between the trained neural network model and the crop, so that when the target image is captured, the crop in the target image can be identified to determine the corresponding neural network. Of course, the user can also select the neural network model corresponding to the crop after knowing the crop to be monitored, and then determine the neural network model to be used for the crop according to the user's selection operation.

After the neural network model is trained, the neural network model can be stored in electronic devices such as drones and control terminals, so that these electronic devices can use the neural network model to perform the crop monitoring method provided by the embodiments of the present application, thereby realizing Growth monitoring of crops.

Please refer to FIG. 6 . FIG. 6 is a schematic flowchart of steps of a method for monitoring the growth of crops provided by an embodiment of the present application. The following is an example of the method for monitoring growth of crops being applied to a control terminal of a flight control system.

As shown in FIG. 6 , the method for monitoring the growth of crops includes steps S201 to S204.

S201 , acquiring a target image obtained by photographing a target area by a drone.

The target area can be the plot that the user needs to monitor, and the target area is planted with crops, such as fruit trees and corn seedlings.

When the user needs to monitor the target area, he can control the drone to fly to the target area and adjust the shooting device of the drone to shoot the target area to obtain a target image, which includes the crops to be monitored.

In some embodiments, the photographing device of the drone can be controlled to photograph the target area from a bird's-eye view, and the target image can be captured from a bird's-eye view, so as to easily identify the estimated radius of crops and multiple growth states. Confidence map of the categories, thereby improving the monitoring efficiency and accuracy of crops.

In some embodiments, the captured target image is an RGB image, or pictures in other formats can be converted into RGB images, and the RGB images are input into the neural network model for learning. RGB images can not only improve the accuracy of model recognition, but also facilitate the combination with the semantic map of the target area to monitor the growth of crops.

In some embodiments, a depth map corresponding to when the drone captures the target image of the crop in the target area may also be obtained, wherein the depth map includes depth information of the crop, and the depth information is the crop distance to the drone.

S202. Input the target image into a pre-trained neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories.

The pre-trained neural network model is obtained by sampling the above model training method, and can learn the target image to output the radius estimation map of the crops in the target image and the confidence maps of multiple growth state categories, such as full convolution Neural network model.

Wherein, the growth state category includes at least one or more of a normal growth category, an abnormal growth category, and a seedling-deficient growth category, and may of course also include other categories. Of course, there are more categories, for example, the abnormal growth category is divided into the category of short growth, the category of insect pests, and so on. Each crop under each growth state category corresponds to a confidence level map, and the confidence level map includes a confidence level value used to represent the current growth state category of the crop, specifically a probability value in [0, 1].

Specifically, as shown in FIG. 7 , the obtained target image is input into the fully convolutional neural network model, the target image is learned through the fully convolutional neural network model, and the radius estimation map of the crops in the target image and multiple Confidence map for growth state categories. Category 1 in FIG. 7 can be a normal growth category, category 2 can be an abnormal growth category, etc., wherein each growth state category corresponds to a confidence map, and the confidence map includes a Gaussian circle of at least one crop and a corresponding radius value, where the Gaussian circle can represent the probability value of the crop being a certain growth state category, such as the probability value of the normal growth category, of course, the confidence map can also include the confidence value. The radius estimate map includes normalized radius values for the crop.

In some embodiments, in order to better count the growth status of crops, the confidence map includes a target position of crops in the confidence map and a confidence value corresponding to the target position, wherein the confidence value is a probability value, which is used to represent the reliability of the category of the growth state of the crop.

Exemplarily, for example, the fully convolutional neural network model is used to detect saplings, and output confidence maps of three categories, which are respectively the confidence map 1 corresponding to the normal growth category, the confidence map 2 and the abnormal growth category. Figure 3. Corresponding confidence levels for the lack of seedling growth categories.

Confidence graph 1 includes the target positions of multiple saplings in the confidence graph and the corresponding confidence values. Similarly, confidence graphs 2 and 3 also include the target positions of multiple saplings in the confidence graph and the corresponding confidence values. , only in the confidence map 2 is the abnormal growth category of saplings, in the confidence map 3 is the corresponding position of the lack of seedlings.

In some embodiments, the target image and the depth map corresponding to the target image may also be input into a pre-trained neural network model, wherein the training data used in training the neural network model includes crops in-depth information. So that the neural network model can learn the target image and the depth map, and output the radius estimation map of the crop and the confidence map of multiple growth state categories. Using the depth map can further improve the accuracy of the output confidence map and radius estimation map.

Specifically, as shown in FIG. 8 , the obtained target image A and the depth map A corresponding to the target image A are input into the fully convolutional neural network model, the target image is learned through the fully convolutional neural network model, and the output Radius estimates for crops in this target image and confidence maps for multiple growth state categories.

In some embodiments, to improve the learning efficiency of the neural network model, before inputting the target image and the depth map corresponding to the target image together into the pre-trained neural network model, the depth map may also be normalized deal with. Specifically, the reference parameters are first determined, and then the depth information of the crops included in the depth map is normalized according to the reference parameters to obtain a normalized depth map, wherein the reference parameters are used as normalized The base number for processing, for example, the depth information of crops is subtracted from the base number.

To determine the reference parameters, specifically, the target crops can be selected from multiple crops in the target image according to the depth information, and the average value of the depth information of the target crops can be used as the base, such as the crops whose depth information is less than a preset threshold. Alternatively, a preset proportion of crops with less depth information is selected as the target crops, and the preset proportion is 1%, that is, the depth information of the selected 1% of the crops is less than the depth information of the remaining 99% of the crops.

S203. Perform information fusion according to the radius estimation map and the confidence maps of the multiple growth state categories to obtain a fusion map.

After obtaining the radius estimation map of the crops in the target image and the confidence maps of multiple growth state categories, the radius estimate map and the confidence maps of multiple growth state categories are fused to obtain a fusion map. The fusion map includes information corresponding to confidence maps and radius estimation maps of multiple growth state categories, thereby facilitating statistics on the growth states of crops according to the fusion map.

In some embodiments, when performing information fusion, the radius estimation graph may be used as a reference graph, and the confidence maps of the multiple growth state categories may be fused and superimposed on the radius estimation graph to obtain a fusion graph. The confidence map is superimposed on the radius estimation map. Specifically, the target position and confidence value of the crops in the confidence map can be added to the corresponding crops in the radius estimate map.

Among them, in order to obtain the fusion map conveniently and quickly, when the confidence maps of multiple growth state categories are fused and superimposed on the radius estimation map, the number of confidence degrees in the confidence map of multiple growth state categories can be specifically calculated. Determine a stacking sequence corresponding to the confidence maps of the multiple growth state categories; and fuse and superimpose the confidence maps of the multiple growth state categories on the radius estimation map according to the stacking sequence.

For example, the number of saplings corresponding to the normal growth category may be relatively large, so the number of corresponding confidence levels is relatively large, while the number of saplings corresponding to the abnormal growth category and the lacking seedling growth category may be relatively small, so the number of corresponding confidence levels is relatively small. Determine the stacking order according to the number of specific confidence levels from the largest number to at least determine the stacking order, that is, the larger number is superimposed first, and the smaller number is superimposed later. Determining the stacking order can easily and quickly obtain a fusion map, thereby improving the analysis efficiency of crops.

In some embodiments, in order to obtain a more accurate analysis effect, before information fusion is performed according to the radius estimation map and the confidence maps of the multiple growth state categories, the confidence map may also be maximized. Value pooling is performed, invalid crops are deleted according to a preset threshold, and valid crops are retained, wherein the pixel values corresponding to the invalid crops are less than or equal to the preset threshold. The preset threshold can be set by the user in order to filter out invalid crops.

When performing the maximum value pooling process, the size of the convolution kernel used is 5*5. Of course, the size of the convolution kernel can also be other values.

S204. Determine the growth status of the crop according to the fusion map.

Specifically, the number proportion of normal growth categories, the number proportion of abnormal growth categories, and the number proportion of seedling-missing growth types in the fusion graph can be counted; alternatively, the positions of crops with abnormal growth categories, and seedling-missing growth categories can also be counted. The corresponding position, etc.; or, the radius value of the crops can also be counted, and whether the overall growth condition of the target area meets the expected requirements can be determined according to the radius value.

Exemplarily, the growth status of the crops is determined according to the fusion map, and specifically, the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories may be counted in the fusion diagram. The quantity of crops and the quantity of crops corresponding to other categories determine the overall growth status of the crops. Other categories are, for example, the abnormal growth category and the lack of seedling growth category.

For example, if the number of saplings corresponding to the normal growth category is 950, and the number of abnormal growth categories and missing seedling growth categories is 50, the proportion of the normal growth category can be calculated, which is 95%. If the user expects a value of 90%, Then it can be determined that the saplings in the target area grow better.

In some embodiments, the semantic map of the target area may be generated according to the plurality of color channel information of the acquired target image, and the semantic map of the target area may be generated according to the plurality of color channel information, and the radius estimation map and the plurality of growth state categories may be used. The confidence map of the semantic map is used to mark the crops in the semantic map. The semantic map can include more abundant information, which is convenient for users to use.

In some embodiments, a semantic map of the target area may also be obtained, wherein the semantic map may be an image obtained by photographing the target area in advance and a map generated by using a semantic segmentation technique on the map. The color areas represent different languages. After acquiring the semantic map, the semantic map can be corrected according to the fusion map to improve the accuracy of the semantic map, or the fusion map can be corrected according to the semantic map to improve the accuracy of the semantic map. The accuracy of the fusion graph. As a result, the accuracy of crop monitoring can be improved, thereby increasing crop yield.

In the method for monitoring the growth of crops provided by the above embodiments, the target image captured by the drone is acquired, and the target image is learned by using the neural network model, so that the growth status of the crops can be accurately monitored, for example, different growth statuses of the crops can be obtained. Category (such as normal growth, abnormal growth or lack of seedlings, etc.), which can facilitate crop administrators to timely and targeted treatment of crops, such as fertilizing, replenishing seedlings or spraying pesticides, etc., which can improve crop yields.

Please refer to FIG. 9 . FIG. 9 is a schematic flow chart of the steps of another method for monitoring the growth of crops provided by the embodiments of the present application. The following is an example of the method for monitoring the growth of crops that can be applied to a control terminal of a flight control system.

As shown in FIG. 9 , the growth monitoring method includes steps S301 to S306.

S301 , acquiring a target image obtained by photographing a target area by a drone.

S302, identifying the type of crops in the target image;

S303, according to the preset correspondence between the type of the crop and the neural network model, determine the neural network model corresponding to the type of the crop in the target image;

S304, inputting the target image into the determined neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories;

S305. Perform information fusion according to the radius estimation map and the confidence maps of the multiple growth state categories to obtain a fusion map; and

S306. Determine the growth status of the crop according to the fusion map.

Specifically, image recognition technology can be used to identify the crops in the target image to determine the type of the crops, such as extracting the characteristics of the crops in the target image, and identifying the types of crops according to the characteristics of the crops. Of course, the crop identification model can also be used to determine the type of crops, and the crop identification type can be obtained by training based on a convolutional neural network. The types of crops can be, for example, grains, vegetables, fruits, medicinal materials, and of course, specific names of crops, such as apple trees, peach trees, corn seedlings, cotton seedlings, and so on.

In order to improve the accuracy of the neural network model, a neural network model can be trained for each crop, that is, different types of crops correspond to different neural network models, and the corresponding relationship between the types of crops and the neural network model can be established to determine the crops. When determining the type, the neural network model corresponding to the type of crops in the target image is determined according to the corresponding relationship, that is, the neural network model to be used.

After the corresponding neural network model is determined, the target image is input into the determined neural network model, and the radius estimation map of the crop and the confidence maps of multiple growth state categories are obtained, so that according to the radius estimation map and all the confidence maps The confidence maps of the above-mentioned multiple growth state categories are fused to obtain a fusion map, and then statistical analysis of the growth state of the crops is performed.

After obtaining the statistical analysis results of the growth conditions of the crops, it is also determined whether the growth conditions of the crops meet the preset growth requirements according to the statistical results. The actual position of the crop to be grown, so that the user can process the crop according to the actual position, and of course, the actual position and growth state category of the crop that does not meet the preset growth requirements can be output, so that the user can carry out targeted processing, thereby improving crop yield.

The preset growth requirements include: the proportion of the number of crops corresponding to the abnormal growth category is greater than the first preset threshold, the proportion of the number of crops corresponding to the lack of seedling growth category is greater than the second preset threshold, and the proportion of the number of crops corresponding to the normal growth category is less than the first preset threshold. One of three preset thresholds. The first preset threshold, the second preset threshold, and the third preset threshold are set according to actual needs, and their sizes are not limited here.

Wherein, the method of determining the actual position of the crops that do not meet the preset growth requirements can be determined by the method of determining the actual position of the crops. Specifically, as shown in FIG. 10, FIG. 10 shows a method for determining the actual position of a crop provided by an embodiment of the present application, and the method includes the following contents:

S401, obtaining the flight information corresponding to the drone when shooting the target image and the field of view angle of the shooting device of the drone when shooting the target image, the flight information including the flight position and the flight height;

S402, obtaining the relative position of the crop in the target image relative to the center point of the target image;

S403. Determine the actual position of the crop according to the flying position, the flying height, the angle of view and the relative position.

When acquiring the target image of the target area captured by the camera of the UAV, the corresponding flight information of the UAV when capturing the target image can also be acquired. The flight information includes at least the flight position and the flight height. The position in the world coordinate system can be measured by GPS on the drone, and the flight height is the distance from the drone to the surface of the crop, which can be measured by radar devices or binocular ranging. At the same time, it is also necessary to obtain the field of view angle of the photographing device when photographing the target image.

If the actual position of a certain crop needs to be determined, it is also necessary to determine the relative position of the crop relative to the center point of the target image in the target image. Therefore, it is convenient to determine the actual position of the crops according to the flight position, flight height, field of view and relative position.

Exemplarily, as shown in Figure 11a and Figure 11b, the drone shoots the target image A from a top-down angle, and the corresponding position of the drone in the target image A is the center point in Figure 11a, and the drone is measured at this time. The flight position is (x ₀ , y ₀ ), the flight height is h, and the field of view of the target image A captured by the photographing device is θ. Assuming that the sapling a is abnormally growing, it is necessary to determine the actual position of the sapling a, so that the user can quickly Identify the sapling a and treat it. Among them, the actual position of the sapling a and the flight position of the drone are both positions in the world coordinate system, so that it is convenient for the user to use GPS for positioning.

When calculating the actual position of the sapling a, it is necessary to first determine the relative position of the sapling a relative to the center point in the target image a, as shown in Figure 11b, the relative position may include the distance d of the sapling a relative to the center point in the target image A and angle α.

After determining the relative position of the sapling a, it is also necessary to determine the actual distance corresponding to the distance d in the world coordinate system, where the distance d is the distance in the target image A. Specifically, the actual distance corresponding to the half-angle width L of the target image A in the world coordinate system can be determined by using the trigonometric function relationship according to the field of view angle θ and the flying height h when the shooting device shoots the target image A; The proportional relationship of the width L in the target image A can determine the actual distance corresponding to the distance d in the world coordinate system.

As shown in Figure 11b, it is assumed that the center point corresponding to the UAV is the coordinate origin O, the distance and angle of the sapling a relative to the center point are the distance d and the angle α, respectively, and the actual distance corresponding to the distance d and the coordinates of the coordinate origin are also determined. (x ₀ , y ₀ ), so the actual position (x ₁ , y ₁ ) of the sapling a can be determined.

In the method for monitoring the growth of crops provided by the above embodiments, the target image captured by the drone is acquired, and the target image is learned by using the neural network model, so that the growth status of the crops can be accurately monitored, for example, different growth statuses of the crops can be obtained. category (such as normal growth, abnormal growth or lack of seedlings, etc.), and can also determine the actual location of crops, so that users can quickly and timely carry out targeted treatments on crops, such as fertilizing, replenishing seedlings or spraying pesticides, etc. , thereby increasing the yield of crops.

Please refer to FIG. 12 , which is a schematic block diagram of an unmanned aerial vehicle provided by an embodiment of the present application. As shown in FIG. 12 , the drone 100 further includes at least one or more processors 101 , a memory 102 and a photographing device 13 .

The processor 101 may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), or the like.

The memory 102 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, or a mobile hard disk, and the like.

The photographing device 13 is used for photographing crops planted in the target area to obtain a target image.

Wherein, the memory 102 is used for storing a computer program; the processor 101 is used for executing the computer program, and when executing the computer program, executes any one of the crop growth monitoring methods provided in the embodiments of this application.

Exemplarily, the processor is configured to execute the computer program and implement the following steps when executing the computer program:

Obtain a target image obtained by photographing a target area by a drone, and the target area is planted with crops; input the target image into a pre-trained neural network model to obtain a radius estimation map and multiple growth states of the crops a confidence map of the categories; perform information fusion according to the radius estimation map and the confidence maps of the multiple growth state categories to obtain a fusion map; and determine the growth state of the crop according to the fusion map.

Referring to FIG. 13 , FIG. 13 is a schematic block diagram of a control terminal provided by an embodiment of the present application. As shown in FIG. 13 , the control terminal 200 further includes at least one or more processors 201 and a memory 202 .

The processor 201 may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), or the like.

The memory 202 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, or a removable hard disk, and the like.

Wherein, the memory 202 is used for storing a computer program; the processor 201 is used for executing the computer program and when executing the computer program, executes any one of the crop growth monitoring methods provided in the embodiments of this application.

Referring to FIG. 14 , FIG. 14 is a schematic block diagram of a monitoring apparatus provided by an embodiment of the present application. As shown in FIG. 14 , the monitoring device 300 further includes at least one or more processors 301 and a memory 302 .

The processor 301 may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), or the like.

The memory 302 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, or a removable hard disk, and the like.

Wherein, the memory 302 is used for storing a computer program; the processor 301 is used for executing the computer program and when executing the computer program, executes any one of the crop growth monitoring methods provided in the embodiments of the present application.

It should be noted that the monitoring apparatus 300 may be configured in an electronic device to implement any one of the methods for monitoring the growth of crops provided in the embodiments of the present application, for example, it may be configured in an unmanned aerial vehicle or a control terminal.

The embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement the above implementation The steps of any one of the methods for monitoring the growth of crops provided in the example.

Wherein, the computer-readable storage medium may be an internal storage unit of the UAV or the control terminal described in any of the foregoing embodiments, such as a memory or memory of the UAV. The computer-readable storage medium can also be an external storage device of the drone, such as a plug-in hard disk equipped on the drone, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in the present application. Modifications or substitutions shall be covered by the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A method for monitoring the growth of crops, characterized in that the method comprises:

obtaining a target image obtained by photographing a target area by a drone, and the target area is planted with crops;

Inputting the target image into a pre-trained neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories;

Perform information fusion according to the radius estimation map and the confidence maps of the plurality of growth state categories to obtain a fusion map; and

The growth status of the crop is determined according to the fusion map.
The method according to claim 1, wherein the growth state categories include at least one or more of a normal growth category, an abnormal growth category, and a seedling-deficient growth category.
The method according to claim 1, wherein the confidence map includes a target position of the crop in the confidence map and a confidence value corresponding to the target position;

Wherein, the confidence value is a probability value, which is used to represent the reliability of the growth state category of the crop.
The method of claim 1, wherein the radius estimation map includes normalized radius values of the crops.
The method according to claim 1, wherein the neural network model comprises a fully convolutional neural network model; and/or,

The target image is an image obtained by photographing the target area by the drone-controlled photographing device from a bird's-eye view; and/or,

The target image input to the neural network model is an RGB image.
The method according to claim 1, wherein the method further comprises:

Obtain the depth map corresponding to when the drone shoots the target image of the crop in the target area, wherein the depth map includes the depth information of the crop, and the depth information is the distance from the crop to the drone ;

The inputting the target image into the pre-trained neural network model includes:

Inputting the target image and the depth map corresponding to the target image together into a pre-trained neural network model;

Wherein, the training data used in training the neural network model includes the depth information of crops.
The method according to claim 6, wherein before the target image and the depth map corresponding to the target image are input together into a pre-trained neural network model, the method further comprises:

determining a reference parameter, wherein the reference parameter is used as a base for normalization;

According to the reference parameters, the depth information of the crops included in the depth map is normalized to obtain a normalized depth map.
The method according to claim 1, wherein the performing information fusion according to the confidence map of the multiple growth state categories and the radius estimation map, comprising:

Using the radius estimation graph as a reference graph, the confidence maps of the multiple growth state categories are fused and superimposed on the radius estimation graph.
The method according to claim 8, wherein the fusion and superposition of the confidence maps of the plurality of growth state categories on the radius estimation map comprises:

determining a stacking sequence corresponding to the confidence maps of the multiple growth state categories according to the number of confidence scores in the confidence maps of the multiple growth state categories; and

The confidence maps of the plurality of growth state categories are fused and superimposed on the radius estimation map according to the superposition order.
The method according to claim 1, wherein before the information fusion is performed according to the radius estimation map and the confidence maps of the multiple growth state categories, the method further comprises:

Perform maximum value pooling on the confidence map, and delete invalid crops according to a preset threshold;

Wherein, the pixel value corresponding to the invalid crop is less than or equal to the preset threshold.
The method according to claim 10, wherein the size of the convolution kernel of the maximum value pooling process is 5*5.
The method according to claim 1, wherein the determining the growth status of the crops according to the fusion map comprises:

Count the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories in the fusion graph,

According to the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories, the overall growth status of the crops is determined.
The method of claim 1, wherein the method comprises:

Obtain the flight information corresponding to the drone when the target image is captured and the field of view of the drone's shooting device when the target image is captured, wherein the flight information includes a flight position and a flight height;

obtaining the relative position of the crop in the target image relative to the center point of the target image;

According to the flying position, the flying height, the angle of view and the relative position, the actual position of the crop is determined.
The method of claim 13, wherein the method comprises:

When it is determined that the growth condition of the crop cannot meet the preset growth requirement, outputting the actual position of the crop;

The preset growth requirements include: the proportion of the number of crops corresponding to the abnormal growth category is greater than the first preset threshold, the proportion of the number of crops corresponding to the lacking seedling growth category is greater than the second preset threshold and the number of crops corresponding to the normal growth category The proportion is less than one of the third preset thresholds.
The method according to any one of claims 1 to 14, wherein the method further comprises:

acquiring sample data, the sample data includes multiple images of the same crop, and the images are marked with multiple growth state categories of the crop and a radius value of the crop;

Divide the sample data into training data and test data;

Based on the determined neural network, model training is performed using the training data, and model testing is performed using the test data, and a neural network model is obtained when the trained model converges as the pre-trained neural network model.
The method of claim 15, wherein the sample data further comprises a depth map of the crop.
The method according to any one of claims 1 to 14, wherein different types of crops correspond to different neural network models;

The inputting the target image into the pre-trained neural network model includes:

identifying the type of crops in the target image;

According to the preset correspondence between the type of crops and the neural network model, determine the neural network model corresponding to the type of crops in the target image;

The target image is input to the determined neural network model.
The method according to any one of claims 1 to 14, wherein the method further comprises:

generating a semantic map of the target area according to a plurality of color channel information of the target image; and

The crops in the semantic map are marked according to the radius estimation map and the confidence maps of the plurality of growth state categories.
The method according to any one of claims 1 to 14, wherein the method further comprises:

obtaining a semantic map of the target area;

The semantic map is corrected according to the fusion map, or the fusion map is corrected according to the semantic map.
An unmanned aerial vehicle, characterized in that the unmanned aerial vehicle comprises:

body;

PTZ, the PTZ is installed on the body;

a photographing device, wherein the photographing device is installed on the cradle, and the photographing angle of the photographing device can be adjusted by adjusting the cradle;

processor and memory;

Wherein, the memory is used to store a computer program and a pre-trained neural network model; the processor is used to execute the computer program and implement the following steps when executing the computer program:

obtaining a target image obtained by photographing a target area by a drone, and the target area is planted with crops;

Inputting the target image into a pre-trained neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories;

Perform information fusion according to the radius estimation map and the confidence maps of the plurality of growth state categories to obtain a fusion map; and

The growth status of the crop is determined according to the fusion map.
The unmanned aerial vehicle according to claim 20, wherein the growth state categories include at least one or more of a normal growth category, an abnormal growth category, and a seedling-deficient growth category.
The drone according to claim 20, wherein the confidence map includes a target position of the crops in the confidence map and a confidence value corresponding to the target position;

Wherein, the confidence value is a probability value, which is used to represent the reliability of the growth state category of the crop.
21. The UAV of claim 20, wherein the radius estimation map includes a normalized radius value of the crop.
The drone according to claim 20, wherein the neural network model comprises a fully convolutional neural network model; and/or,

The target image is an image obtained by photographing the target area by the drone-controlled photographing device from a bird's-eye view; and/or,

The target image input to the neural network model is an RGB image.
The drone of claim 20, wherein the processor is configured to:

Obtain the depth map corresponding to when the drone shoots the target image of the crop in the target area, wherein the depth map includes the depth information of the crop, and the depth information is the distance from the crop to the drone ;

The inputting the target image into the pre-trained neural network model includes:

Inputting the target image and the depth map corresponding to the target image together into a pre-trained neural network model;

Wherein, the training data used in training the neural network model includes the depth information of crops.
The drone according to claim 25, wherein before the target image and the depth map corresponding to the target image are input together into a pre-trained neural network model, the processor is configured to:

determining a reference parameter, wherein the reference parameter is used as a base for normalization;

According to the reference parameters, the depth information of the crops included in the depth map is normalized to obtain a normalized depth map.
The unmanned aerial vehicle according to claim 20, wherein the performing information fusion according to the confidence map of the multiple growth state categories and the radius estimation map, comprising:

Using the radius estimation graph as a reference graph, the confidence maps of the multiple growth state categories are fused and superimposed on the radius estimation graph.
The unmanned aerial vehicle according to claim 27, wherein the fusion and superposition of the confidence maps of the plurality of growth state categories on the radius estimation map, comprising:

determining a stacking sequence corresponding to the confidence maps of the multiple growth state categories according to the number of confidence scores in the confidence maps of the multiple growth state categories; and

The confidence maps of the plurality of growth state categories are fused and superimposed on the radius estimation map according to the superposition order.
The UAV according to claim 20, wherein before the information fusion is performed according to the radius estimation map and the confidence maps of the multiple growth state categories, the processor is configured to:

Perform maximum value pooling on the confidence map, and delete invalid crops according to a preset threshold;

Wherein, the pixel value corresponding to the invalid crop is less than or equal to the preset threshold.
The drone according to claim 29, wherein the size of the convolution kernel of the maximum value pooling process is 5*5.
The unmanned aerial vehicle according to claim 20, wherein the determining the growth status of the crops according to the fusion map comprises:

Count the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories in the fusion graph,

The overall growth status of the crops is determined according to the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories.
The drone of claim 20, wherein the processor is configured to:

Obtain the flight information corresponding to the drone when the target image is captured and the field of view of the drone's shooting device when the target image is captured, wherein the flight information includes a flight position and a flight height;

obtaining the relative position of the crop in the target image relative to the center point of the target image;

According to the flying position, the flying height, the angle of view and the relative position, the actual position of the crop is determined.
The drone of claim 32, wherein the processor is configured to:

When it is determined that the growth condition of the crop cannot meet the preset growth requirement, outputting the actual position of the crop;

Wherein, the preset growth requirements include: the proportion of the number of crops corresponding to the abnormal growth category is greater than the first preset threshold, the proportion of the number of crops corresponding to the lacking seedling growth category is greater than the second preset threshold and the number of crops corresponding to the normal growth category The proportion is less than one of the third preset thresholds.
The drone according to any one of claims 20 to 33, wherein the processor is configured to implement:

acquiring sample data, the sample data includes multiple images of the same crop, and the images are marked with multiple growth state categories of the crop and a radius value of the crop;

Divide the sample data into training data and test data;

Based on the determined neural network, model training is performed using the training data, and model testing is performed using the test data, and a neural network model is obtained when the trained model converges as the pre-trained neural network model.
35. The drone of claim 34, wherein the sample data further comprises a depth map of the crop.
The drone according to any one of claims 20 to 33, wherein different types of crops correspond to different neural network models;

The inputting the target image into the pre-trained neural network model includes:

identifying the type of crops in the target image;

According to the preset correspondence between the type of crops and the neural network model, determine the neural network model corresponding to the type of crops in the target image;

The target image is input to the determined neural network model.
The drone according to any one of claims 20 to 33, wherein the processor is configured to implement:

generating a semantic map of the target area according to a plurality of color channel information of the target image; and

The crops in the semantic map are marked according to the radius estimation map and the confidence maps of the plurality of growth state categories.
The drone according to any one of claims 20 to 33, wherein the processor is configured to implement:

obtaining a semantic map of the target area;

The semantic map is corrected according to the fusion map, or the fusion map is corrected according to the semantic map.
A control terminal, characterized in that the control terminal is connected to the unmanned aerial vehicle in communication for controlling the flight of the unmanned aerial vehicle, and the unmanned aerial vehicle comprises a pan/tilt and a camera mounted on the pan/tilt. device; the control terminal includes:

processor and memory;

Wherein, the memory is used to store a computer program and a pre-trained neural network model; the processor is used to execute the computer program and implement the following steps when executing the computer program:

obtaining a target image obtained by photographing a target area by a drone, and the target area is planted with crops;

Inputting the target image into a pre-trained neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories;

Perform information fusion according to the radius estimation map and the confidence maps of the plurality of growth state categories to obtain a fusion map; and

The growth status of the crop is determined according to the fusion map.
The control terminal according to claim 39, wherein the growth state categories include at least one or more of a normal growth category, an abnormal growth category, and a seedling-deficient growth category.
The control terminal according to claim 39, wherein the confidence map includes a target position of the crop in the confidence map and a confidence value corresponding to the target position;

Wherein, the confidence value is a probability value, which is used to represent the reliability of the growth state category of the crop.
The control terminal according to claim 39, wherein the radius estimation map includes a normalized radius value of the crop.
The control terminal according to claim 39, wherein the neural network model comprises a fully convolutional neural network model; and/or,

The target image is an image obtained by photographing the target area by the drone-controlled photographing device from a bird's-eye view; and/or,

The target image input to the neural network model is an RGB image.
The control terminal according to claim 39, wherein the processor is configured to implement:

Obtain the depth map corresponding to when the drone shoots the target image of the crop in the target area, wherein the depth map includes the depth information of the crop, and the depth information is the distance from the crop to the drone ;

The inputting the target image into the pre-trained neural network model includes:

Inputting the target image and the depth map corresponding to the target image together into a pre-trained neural network model;

Wherein, the training data used in training the neural network model includes the depth information of crops.
The control terminal according to claim 44, wherein before the target image and the depth map corresponding to the target image are input together into a pre-trained neural network model, the processor is configured to:

determining a reference parameter, wherein the reference parameter is used as a base for normalization;

According to the reference parameters, the depth information of the crops included in the depth map is normalized to obtain a normalized depth map.
The control terminal according to claim 39, wherein the performing information fusion according to the confidence map of the plurality of growth state categories and the radius estimation map comprises:

Using the radius estimation graph as a reference graph, the confidence maps of the multiple growth state categories are fused and superimposed on the radius estimation graph.
The control terminal according to claim 46, wherein the fusion and superposition of the confidence maps of the multiple growth state categories on the radius estimation map comprises:

determining a stacking sequence corresponding to the confidence maps of the multiple growth state categories according to the number of confidence scores in the confidence maps of the multiple growth state categories; and

The confidence maps of the plurality of growth state categories are fused and superimposed on the radius estimation map according to the superposition order.
The control terminal according to claim 39, wherein before the information fusion is performed according to the radius estimation map and the confidence maps of the multiple growth state categories, the processor is configured to:

Perform maximum value pooling on the confidence map, and delete invalid crops according to a preset threshold;

Wherein, the pixel value corresponding to the invalid crop is less than or equal to the preset threshold.
The control terminal according to claim 48, wherein the size of the convolution kernel of the maximum value pooling process is 5*5.
The control terminal according to claim 39, wherein the determining the growth status of the crops according to the fusion map comprises:

Count the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories in the fusion graph,

The overall growth status of the crops is determined according to the number of crops corresponding to the normal growth category and the number of crops corresponding to other categories.
The control terminal according to claim 39, wherein the processor is configured to implement:

Obtain the flight information corresponding to the drone when the target image is captured and the field of view of the drone's shooting device when the target image is captured, wherein the flight information includes a flight position and a flight height;

obtaining the relative position of the crop in the target image relative to the center point of the target image;

According to the flying position, the flying height, the angle of view and the relative position, the actual position of the crop is determined.
The control terminal according to claim 51, wherein the processor is configured to implement:

When it is determined that the growth condition of the crop cannot meet the preset growth requirement, outputting the actual position of the crop;

Wherein, the preset growth requirements include: the proportion of the number of crops corresponding to the abnormal growth category is greater than the first preset threshold, the proportion of the number of crops corresponding to the lacking seedling growth category is greater than the second preset threshold and the number of crops corresponding to the normal growth category The proportion is less than one of the third preset thresholds.
The control terminal according to any one of claims 39 to 52, wherein the processor is configured to implement:

acquiring sample data, the sample data includes multiple images of the same crop, and the images are marked with multiple growth state categories of the crop and a radius value of the crop;

Divide the sample data into training data and test data;

Based on the determined neural network, model training is performed using the training data, and model testing is performed using the test data, and a neural network model is obtained when the trained model converges as the pre-trained neural network model.
The control terminal according to claim 53, wherein the sample data further comprises a depth map of the crops.
The control terminal according to any one of claims 39 to 52, wherein different types of crops correspond to different neural network models;

The inputting the target image into the pre-trained neural network model includes:

identifying the type of crops in the target image;

According to the preset correspondence between the type of crops and the neural network model, determine the neural network model corresponding to the type of crops in the target image;

The target image is input to the determined neural network model.
The control terminal according to any one of claims 39 to 52, wherein the processor is configured to implement:

generating a semantic map of the target area according to a plurality of color channel information of the target image; and

The crops in the semantic map are marked according to the radius estimation map and the confidence maps of the plurality of growth state categories.
The control terminal according to any one of claims 39 to 52, wherein the processor is configured to implement:

obtaining a semantic map of the target area;

The semantic map is corrected according to the fusion map, or the fusion map is corrected according to the semantic map.
A monitoring device, characterized in that the monitoring device comprises:

processor and memory;

Wherein, the memory is used to store a computer program and a pre-trained neural network model; the processor is used to execute the computer program and implement the following steps when executing the computer program:

obtaining a target image obtained by photographing a target area by a drone, and the target area is planted with crops;

Inputting the target image into a pre-trained neural network model to obtain a radius estimation map of the crop and a confidence map of multiple growth state categories;

Perform information fusion according to the radius estimation map and the confidence maps of the plurality of growth state categories to obtain a fusion map; and

The growth status of the crop is determined according to the fusion map.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor can realize the implementation of any one of claims 1 to 19. The steps of the growth monitoring method.