CN111967523B - Data fusion agricultural condition detection system and method based on multi-rotor aircraft - Google Patents

Abstract

The invention discloses a data agricultural condition detection system based on a multi-rotor aircraft, which comprises: a data fusion unit; the digital imaging radar unit is electrically connected with the data fusion unit and is used for providing a point cloud picture for the data fusion unit; the multispectral camera unit is electrically connected with the data fusion unit and is used for providing a visible light image for the data fusion unit; the millimeter radar wave unit is electrically connected with the data fusion unit and is used for measuring the relative height between the millimeter radar wave unit and the ground; and the execution unit is in bidirectional electric connection with the data fusion unit and is used for adjusting the spraying dosage of the pesticide. The density condition of crops can be evaluated, and the pesticide spraying amount can be adjusted. The invention further provides a data agricultural condition detection method based on the multi-rotor aircraft.

Description

A data fusion agricultural situation detection system and method based on multi-rotor aircraft

technical field

The invention relates to a multi-rotor aircraft-based data fusion agricultural situation detection system and method, belonging to the field of agricultural informatization.

Background technique

In recent years, with the third wave of the information industry, the Internet of Things technology has developed vigorously. The development of many agricultural fields such as mechanical operation service scheduling. The Internet of Things technology feeds back and processes various data collected by sensors placed in the field, and realizes real-time agricultural monitoring and data collection and transmission to a certain extent.

For the collection and processing of agricultural inspection data in the field of plant protection, there are currently two main methods widely used in the market. One is to use drone aerial surveys before plant protection flight operations, and use multi-spectral cameras to collect approximate crop growth status. Path planning; the other is to use sensors to sense the terrain during the flight, so as to adjust the flight height in a single way. From the perspective of product technology, the data collected by a single sensor has not been processed by data fusion, and only the way of controlling a single variable cannot achieve complete agricultural monitoring and precise pesticide application; from the perspective of product performance, related products have not focused on The functions of comprehensive agricultural monitoring and plant protection drones need to be further improved.

Contents of the invention

The invention designs and develops a data fusion agricultural situation detection system based on multi-rotor aircraft, which can overcome the problem of seedling burning caused by traditional plant protection drones that cannot adjust the amount of spraying in real time according to the operating terrain, sparseness of plant distribution, and crop growth. .

The invention also designs and develops a multi-rotor aircraft-based data fusion agricultural situation detection method, which performs data fusion based on point cloud images and multispectral images, evaluates the density of crops, and realizes the adjustment of pesticide spraying dosage.

The technical scheme provided by the invention is:

A data fusion agricultural situation detection system based on multi-rotor aircraft, including:

data fusion unit;

A digital imaging radar unit, which is electrically connected to the data fusion unit, and is used to provide a point cloud image to the data fusion unit;

a multispectral camera unit, electrically connected to the data fusion unit, for providing visible light images to the data fusion unit;

A millimeter radar wave unit, which is electrically connected to the data fusion unit, is used to measure the relative height to the ground;

The execution unit is electrically connected to the data fusion unit bidirectionally, and is used for adjusting the spraying dosage of the pesticide.

Preferably, the digital imaging radar unit comprises:

imaging radar;

a first storage unit, which is bidirectionally electrically connected to the imaging radar;

The first processing unit is bidirectionally connected to the first storage unit and electrically connected to the data fusion unit.

Preferably, the multispectral camera unit includes:

multispectral camera;

A second storage unit, which is electrically connected to the multispectral camera bidirectionally;

The second processing unit is bidirectionally electrically connected to the second storage unit and electrically connected to the data fusion unit.

Preferably, the data fusion unit includes:

a third storage unit electrically connected to the multispectral camera unit and the digital imaging radar unit simultaneously;

A third processing unit is electrically connected bidirectionally to the third storage unit and electrically connected to the execution structure.

A multi-rotor aircraft-based data fusion agricultural situation detection method, using the multi-rotor aircraft-based data fusion agricultural situation detection system, and includes:

Determine and maintain the flying height between the UAV and the ground, and send the point cloud image to the data fusion unit after sector scanning under the UAV;

Take photos and analysis of the drone below, and send visible light images to the data fusion unit;

Data fusion of point cloud images and multispectral images is performed to evaluate the density distribution of crops and sent to the executive agency to adjust the amount of pesticide spraying in real time to achieve real-time variable spraying.

Preferably, the data fusion process includes:

Set the point cloud image as image A, and the visible light image as image B;

Decompose the image A and the image B on the L-level scale respectively, and fuse the two sets of decomposition coefficients according to the fusion strategy to obtain the fusion coefficient;

The fused image is obtained by inverse reconstruction of the decomposition coefficients after multi-scale inverse fusion;

Carry out image enhancement, judge the density of crops by evaluating the area of the highlighted area, and then adjust the liquid flow.

Preferably,

The decomposition coefficient formula of the image A at the i scale is:

The decomposition coefficient formula of the image B at the i scale is:

The fusion coefficient formula is:

Among them, i=1,2,...,L, F is the coefficient obtained after multi-scale inversion, and L is the scale number of decomposition.

Preferably, the image enhancement includes:

Step 1, performing statistics on the pixel grayscale in the histogram of the fusion image, and obtaining the number of idle grayscales whose pixel number is zero in the RGB color grayscale range [0-255];

Modify the image histogram to:

Count the number of gray levels with his'=0 to obtain the number of gray levels L _r ;

In the formula, his[r] is the data to be processed, Δ is the threshold, and his'[r] is the processed data;

Step 2. Calculate the number L _e of effective gray levels within the entire gray range [0,255],

L _e =255-L _r ,

Step 3. Assign the idle gray level to the effective gray level, and perform nonlinear stretching transformation on the effective gray level that is not zero in the entire gray scale range. The transformation function is:

In the formula, S _k is the enhanced gray level, T is the gray level coefficient, r _k is the pixel area to be processed, and L' _i is the pixel gray level of the image to be processed;

By calculating the area of the highlighted area, adjust the liquid flow L in real time. The empirical formula for flow adjustment is:

L= _Sk ×P;

In the formula, P is the basic operation flow.

The beneficial effect of the present invention is to solve the problem that traditional plant protection drones cannot adjust the amount of spraying in real time according to the operating terrain, the sparseness of plant distribution, and the growth of crops, resulting in burning seedlings. Traditional plant protection UAVs use millimeter-wave radar. Through the data collected by the radar installed under the flying plant protection UAV, the relative distance between the plant protection UAV and the ground is judged to ensure the relative distance between the plant protection UAV and the ground. The height remains constant at all times, which solves the problem that the plant protection drone can fly safely during operation and cannot realize the real-time adjustment of the amount of spraying liquid according to the growth of crops through real-time sensing of the surrounding environment.

Description of drawings

FIG. 1 is a schematic diagram of the overall structure of the multi-rotor aircraft-based data fusion agricultural situation detection system according to the present invention.

Fig. 2 is a schematic structural diagram of the digital radar imaging unit of the present invention.

Fig. 3 is a schematic structural diagram of a multispectral camera unit according to the present invention.

Fig. 4 is a schematic structural diagram of a data fusion unit according to the present invention.

Fig. 5 is a schematic structural diagram of the actuator according to the present invention.

FIG. 6 is a schematic structural diagram of a millimeter wave radar unit according to the present invention.

Fig. 7 is a flowchart of the fusion algorithm described in the present invention.

Fig. 8 is a schematic diagram of a simplified area beamformer according to the present invention.

Fig. 9 is a block diagram of the neural network PID according to the present invention.

Fig. 10 is a general PID step response diagram according to the present invention.

Fig. 11 is a step response diagram of the neural network PID according to the present invention.

Fig. 12 is a flowchart of the variable spraying system of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings, so that those skilled in the art can implement it with reference to the description.

As shown in Figures 1-12, the present invention provides a data fusion agricultural situation detection system based on a multi-rotor aircraft, including: a data fusion unit; a digital imaging radar unit, a multispectral camera unit, an execution unit, and a millimeter wave radar unit.

The digital imaging radar unit is electrically connected to the data fusion unit for providing point cloud images to the data fusion unit, the multispectral camera unit is electrically connected to the data fusion unit for providing visible light images to the data fusion unit, and the millimeter wave radar unit is electrically connected to the data fusion unit, It is used to measure the relative height between the system and the ground; the actuator is electrically connected to the data fusion unit, and is used to adjust the spraying amount of pesticide in real time.

The plant protection UAV is used as an actuator in the actual operation flight, and the millimeter wave radar unit is used to send millimeter wave signals to the ground. Control system to control the plant protection UAV to maintain a relatively stable flight height with the ground; use the digital imaging radar unit to scan the underside of the plant protection UAV in flight and collect the returned data, and finally generate a point cloud image and provide it to Data fusion unit; through the multi-spectral camera, take pictures and analyze the underside of the flying plant protection drone, generate visible light images in different intervals and provide them to the data fusion unit. The data fusion unit performs data fusion based on point cloud images and multi-spectral images, evaluates the density distribution of crops, and sends it to the executive agency to adjust the amount of pesticide spraying in real time to achieve real-time variable spraying of pesticides.

The digital imaging radar unit includes: imaging radar, a first storage unit and a first processing unit, the first storage unit is electrically connected to the imaging radar and the first processing unit in both directions, the first processing unit is electrically connected to the data fusion unit, the first storage unit The unit is used to store the raw data collected by the imaging radar and the control data of the imaging radar by the control unit. The first processing unit is used to process the raw data collected by the imaging radar unit to perform operations and generate point cloud images, and transmit the image data to the data fusion unit.

As shown in Figure 7, the digital imaging radar unit uses phased array radar as the emission source. The main hardware system consists of a high-speed signal processing board, including 8 channels of 10bits AD converters, an FPGA, and a 10hitsDA conversion chip. In this system, we use a combination of hardware and software. In the hardware part, each channel performs data measurement under the control of FPCA, and completes the analog-to-digital conversion of the medium signal sent by the SHA analog intermediate frequency input interface, forming 8 data streams and sending them to the FPGA, and completing digital quadrature in the FPGA Demodulation, 8-channel single-beam DBP processing, and pulse compression processing form a dual-channel digital I/Q signal. After modulo calculation, a single-channel digital video signal is formed and sent to the DA chip.

In the hardware structure of digital imaging radar, the radio frequency (RF) signal received by each unit of the array antenna is converted from analog to digital by the A/D converter in the respective processing module, and then processed by down-conversion and synchronous detection. The orthogonal baseband signal is sent to the DBF processor, and the complex baseband digital signal S and the predetermined complex weight vector W are multiplied and accumulated in the DBF generator, and then the required point cloud data output is obtained.

The multi-spectral camera unit includes: a multi-spectral camera, a second storage unit and a second processing unit, and the second storage unit is electrically connected to the multi-spectral camera and the second processing unit in both directions. The second storage unit is used to store the original data collected by the multi-spectral camera and the control data of the multi-spectral camera by the control unit, and the second processing unit is used to process the original data collected by the multi-spectral camera to perform operations and generate images of different visible light bands. The image data is passed to the data fusion unit.

When working, the light passes through the aperture, and the aperture and focal length are adjusted to ensure appropriate luminous flux and image clarity; then the light passes through the optical extender, and the imaging range is tuned every 5nm by LCTF liquid crystal light splitting; again, the light passes through the converter The adapter is connected to the focal plane of the CMOs area array detector for imaging, and the CMOS detector sequentially transmits the collected images to the computer for storage. Finally, the computer analyzes and processes the collected multispectral images.

The data fusion unit includes a third storage unit and a third processing unit, the third storage unit is electrically connected to the multispectral camera unit and the digital imaging radar unit at the same time, and is electrically connected to the third processing unit bidirectionally, and the signal is transmitted through the third processing unit Give the execution structure. The third storage unit is used to store the point cloud image data generated by the imaging radar unit and the image data of different visible light bands produced by the multispectral camera unit; the third processing unit is used to process the point cloud image data generated by the imaging radar unit and the multispectral camera The image data of different visible light bands generated by the unit is synthesized into an image and evaluated by the algorithm, and finally the evaluation result is passed to the executive agency, which adjusts the spraying operation in real time according to the operating terrain, the sparseness of plant distribution and the growth of crops Dosage.

The present invention also provides a multi-rotor flying chess-based data fusion agricultural situation monitoring method, which performs data fusion based on point cloud images and multispectral images, evaluates the density of crops, and realizes the adjustment of pesticide spraying dosage, including:

Determine and maintain the flying height between the UAV and the ground, and send the point cloud image to the data fusion unit after sector scanning under the UAV;

Take photos and analysis of the drone below, and send visible light images to the data fusion unit;

Data fusion of point cloud images and multispectral images is performed to evaluate the density distribution of crops and sent to the executive agency to adjust the amount of pesticide spraying in real time to achieve real-time variable spraying.

Describe the image processing procedure. Image fusion can provide more effective information for image segmentation, target detection and recognition, image understanding, etc. According to the characteristics of the processed multi-spectral images and point cloud images, this paper uses pixel-level fusion to fuse the collected original image data, and then analyzes and processes the fused image data. The fused image contains the distribution of plant branches (from the point cloud image) and the density information of plant leaves.

Image fusion technology based on multi-scale analysis is a research hotspot in the field of image fusion. It adopts the way of human vision to perceive the objective world from "coarse" to "fine", and can obtain good fusion results. The multi-scale decomposition method of the image obtains the decomposition components in different scale spaces of the image, thereby separating the high and low frequency information in the image, which is similar to the way the human eye processes visual signals. Multi-scale decomposition methods are widely used in various fields of image processing, in which the separation of image edge details and global approximation information obtains computational convenience and reliability. In order to use the information characteristics in different levels of the image to realize the evaluation, selection and fusion of image information flexibly and quickly, the image fusion method used in this study is a fusion method based on multi-scale decomposition. The process is as follows:

As shown in Figure 7, set the point cloud image as image A, and the visible light image as image B;

1) In order to obtain the respective multi-scale decomposition coefficients, the input image A and image B are respectively decomposed into L-level scales,

The decomposition coefficient formula of image A at scale i is:

The decomposition coefficient formula of image B at scale i is:

2) According to the fusion strategy, the two sets of decomposition coefficients are fused to obtain the fusion coefficients;

The fusion coefficient formula is:

3) Then use multi-scale inverse transformation to reversely reconstruct the fused image from the fused decomposition coefficients to obtain:

Wherein, i=1,2,...,L, F is the coefficient obtained after transformation, and L is the scale number of decomposition;

Image enhancement methods can be divided into spatial domain and frequency domain based methods. The enhancement methods based on the spatial domain mainly include histogram specification, histogram equalization method, gray scale transformation method and unsharp mask method, etc. The histogram equalization method enhances all the pixels in the image, but its disadvantage is that it is not easy to highlight the target in the image. In theory, the histogram specification can enhance specific information in the image, but it is difficult to choose an optimal histogram in practical applications. The histogram stretching method can enhance the image contrast to a certain extent, but the change function must be determined by experience or experiments, and this method is subject to greater constraints in application. The enhancement method based on the frequency domain is mainly used for image noise removal and edge enhancement.

We use an adaptive image enhancement algorithm based on nonlinear stretching to enhance the image, which avoids combining different levels of gray levels. Compared with other linear stretching and nonlinear stretching algorithms, the determination of transformation functions and adjustment parameters does not require With the help of experience or experiments, the calculation amount of the algorithm is not small, the self-adaptation is strong, and it is suitable for real-time processing system.

Specifically include:

Step 1: Count the pixel gray levels in the original image histogram to obtain the number of free gray levels with zero pixels in [0-255], [0-255] is the definition of gray levels in RGB colors, from all white to The definition range of full black is [0-255]. Theoretically, the gray level with an absolute frequency of zero is the idle gray level, but the imaging device is not an ideal device. There is a certain proportion of noise points in the image. We will show the gray level as Gray levels with a frequency less than a certain threshold are classified as idle gray levels, which reduces the interference of noise gray levels with small frequencies in idle gray statistics. In this way, we obtain the highest image signal-to-noise ratio, improve the contrast and human visual effect, that is, modify the image histogram to:

The number of gray levels with statistics his'=0 is L _r

Count the number of gray levels with his'=0 to obtain the number of gray levels L _r ;

In the formula, his[r] is the data to be processed, Δ is the threshold, and his'[r] is the processed data

Step 2. Calculate the number L _e of effective gray levels within the entire gray range [0,255],

L _e =255-L _r ;

Step 3. Allocate idle gray levels to valid gray levels. Gray levels with lower frequency of occurrence, that is, fewer pixels, are allocated to more idle gray levels, and gray values with higher frequency of occurrence are allocated less. This is equivalent to nonlinear stretching of the histogram on the gray axis, that is, the smaller the frequency, the greater the stretching distance, and the larger the gray level spacing, the smaller it is. In this way, the interval of the gray levels of the target details with a small frequency of occurrence is stretched, and the details are enhanced. In the histogram equalization method, the target segment with low frequency of gray level is prevented from being degenerated by the background segment with high frequency of gray level;

Step 4. Perform non-linear stretching transformation on the effective gray scale that is not zero in the entire gray scale range, and the transformation function is:

In the formula, S _k is the enhanced gray level, T is the gray level coefficient, r _k is the pixel area to be processed, and L' _i is the pixel gray level of the image to be processed;

The gray scale after image processing is equal to the area gray scale multiplied by the gamma coefficient, and the calculation result of each pixel in the area and the gray scale coefficient. By allocating idle gray levels to effective gray levels, gray levels with smaller frequency of occurrence, that is, fewer pixels, are allocated to more idle gray levels, and gray values with higher frequency of occurrence are allocated less. This is equivalent to nonlinear stretching of the histogram on the gray axis, that is, the smaller the frequency, the greater the stretching distance, and the larger the gray level spacing, the smaller it is. In this way, the interval of the gray levels of the target details with a small frequency of occurrence is stretched, and the details are enhanced.

As a result, an enhanced image is obtained, and the branches and leaves are highlighted, and the liquid flow is adjusted in real time by calculating the area of the highlighted area.

By calculating the area of the highlighted area, adjust the liquid flow L in real time. The empirical formula for flow adjustment is:

As shown in Figure 9-11, the design structure of the PID control algorithm for liquid pumps based on neural network machine learning is shown in Figure 8. In the model using neural network algorithms, two neural network modules with different functions are used to assume different responsibilities , one is the NNI online identifier, and the other is the NNC adaptive PID control processor. The working principle of the flow controller of the liquid pump in agricultural plant protection is: use the algorithm model to adjust the weight system of the identification result of the NNC controlled sub-item in the algorithm in real time, so that the controlled item has self-adaptability and stability sex. In the later period, the stability and performance of the neural network PID control system were verified by Matlab computing software, and a large number of data simulation verifications were carried out. The experimental results are shown in Figure 9-10. It is not difficult to see that the PID control system based on the neural network has better control characteristics than the traditional control system.

As shown in Figure 11, for the development of a complete control system, the hardware is only a part of the development, and the quality of the software directly affects the realization of the overall system function. The control system adopts modular programming, and the whole is realized by writing C language code. In this study, FreeRTOS is selected as the core of the software system, and the Keil integrated development environment is used to complete the software modularization.

Adjust the liquid flow in real time by calculating the area of the high-brightness area:

L= _Sk ×P;

In the formula, P is the basic operation flow.

The flow rate is equal to the enhanced gray level of the current area multiplied by the current crop type. The basic operation flow P is manually determined according to the type of crops in the operation field. The system obtains the enhanced gray level through the image enhancement algorithm based on the image of the crops captured in the operation area. The denser the crops, the higher the gray level S _k , that is, the required The greater the operating flow rate, the real-time variable spraying is realized.

Although the embodiment of the present invention has been disclosed as above, it is not limited to the use listed in the specification and implementation, it can be applied to various fields suitable for the present invention, and it can be easily understood by those skilled in the art Therefore, the invention is not limited to the specific details and examples shown and described herein without departing from the general concept defined by the claims and their equivalents.

Claims (5)

1. A data fusion agricultural condition detection method based on a multi-rotor aircraft is characterized in that a data fusion agricultural condition detection system based on the multi-rotor aircraft is used for detection, and the data fusion agricultural condition detection system based on the multi-rotor aircraft comprises:

a data fusion unit;

the digital imaging radar unit is electrically connected with the data fusion unit and is used for providing a point cloud picture for the data fusion unit;

the multispectral camera unit is electrically connected with the data fusion unit and is used for providing a visible light image for the data fusion unit;

the millimeter radar wave unit is electrically connected with the data fusion unit and is used for measuring the relative height between the millimeter radar wave unit and the ground;

the execution unit is electrically connected with the data fusion unit in a bidirectional way and is used for adjusting the spraying amount of the pesticide;

the data fusion agricultural condition detection method based on the multi-rotor aircraft comprises the following steps:

determining and maintaining the flying height of the unmanned aerial vehicle and the ground, and sending a cloud point map to the data fusion unit after sector scanning is carried out below the unmanned aerial vehicle;

photographing and analyzing the lower part of the unmanned aerial vehicle, and sending a visible light image to the data fusion unit;

performing data fusion on the point cloud image and the multispectral image, evaluating the density distribution of crops and sending the density distribution to an executing mechanism to adjust the spraying amount of pesticides in real time so as to realize real-time variable spraying;

the data fusion process comprises the following steps:

setting a point cloud picture as an image A and setting a visible light image as an image B;

respectively carrying out L-layer scale decomposition on the image A and the image B, and fusing the two groups of decomposition coefficients according to a fusion strategy to obtain fusion coefficients;

reversely reconstructing by adopting the decomposition coefficient after the multi-scale inverse transformation fusion to obtain a fused image;

performing image enhancement, judging the density of crops by evaluating the area of the highlight area, and further adjusting the flow of the liquid medicine;

the digital imaging radar unit includes:

an imaging radar;

a first storage unit which is electrically connected with the imaging radar in a bidirectional way;

and the first processing unit is in bidirectional point connection with the first storage unit and is electrically connected with the data fusion unit.

2. The multi-rotor aircraft-based data fusion agricultural condition detection method according to claim 1, wherein the multispectral camera unit comprises:

a multispectral camera;

a second storage unit which is electrically connected with the multispectral camera in a bidirectional way;

and the second processing unit is bidirectionally and electrically connected with the second storage unit and is electrically connected with the data fusion unit.

3. The multi-rotor aircraft-based data fusion agricultural condition detection method according to claim 2, wherein the data fusion unit comprises:

a third storage unit electrically connected to the multispectral camera unit and the digital imaging radar unit at the same time;

and the third processing unit is bidirectionally and electrically connected with the third storage unit and is electrically connected with the execution mechanism.

4. The multi-rotor aircraft-based data fusion agricultural condition detection method according to claim 3,

the decomposition coefficient formula of the image A at the i scale is as follows:

the decomposition coefficient formula of the image B at i scale is as follows:

the fusion coefficient formula is as follows:

wherein, i =1,2, the.

5. The multi-rotor aircraft-based data fusion agricultural condition detection method of claim 4, wherein the image enhancement comprises:

step 1, counting the pixel gray in the histogram of the fusion image to obtain the number of idle gray with zero pixel number in the RGB color gray range of [0-255 ];

modifying the image histogram to:

counting the number of the gray levels of his' =0 to obtain the number of the gray levels L _r ；

In the formula, his r is the data to be processed, delta is the threshold value, and his' r is the processed data;

step 2, calculating the whole gray scale range [0,255 ]]Number of inner effective gray levels L _e ，

L _e ＝255-L _r ，

Step 3, assigning the idle gray level to an effective gray level, and performing nonlinear stretching transformation on the effective gray level which is not zero in the whole gray level range, wherein the transformation function is as follows:

in the formula, S _k For enhanced gray scale, T is the gray scale coefficient, r _k Is a pixel region requiring processing of L' _i The gray scale of the pixel of the image to be processed;

the liquid medicine flow L is adjusted in real time by calculating the area of the highlight area, and the empirical formula of the flow adjustment is as follows:

L＝S _k ×P；

wherein P is the basic operation flow.

Images