CN116883480A

CN116883480A - Corn plant height detection method based on binocular image and ground-based radar fusion point cloud

Info

Publication number: CN116883480A
Application number: CN202310867031.6A
Authority: CN
Inventors: 李丹; 王根; 王冬; 段佳琪; 王红; 杨宇航; 徐瑞婷; 高孟利
Original assignee: Northeast Forestry University
Current assignee: Northeast Forestry University
Priority date: 2023-07-14
Filing date: 2023-07-14
Publication date: 2023-10-13

Abstract

A corn plant height detection method based on binocular image and ground-based radar fusion point cloud relates to the technical field of agricultural detection, and aims at solving the problems that in the prior art, point cloud data which is reconstructed through binocular image three-dimensional reconstruction and point cloud data which is acquired through ground-based radar are both in the absence of point cloud information at different positions, so that the precision of single corn plant height detection is reduced, and when corn plant height detection is carried out, the point cloud data which is reconstructed through binocular image three-dimensional reconstruction and the point cloud data which is acquired through ground-based radar are both in the absence of point cloud information at different positions, so that the accuracy of detection results is greatly affected. In order to solve the problem, the application provides a data fusion method, which fuses data acquired by a binocular camera and a ground-based radar on a point cloud layer surface, so that the point cloud information loss of the binocular camera and the ground-based radar at different positions is mutually compensated, and the accuracy of single-plant corn plant height detection is improved.

Description

Corn plant height detection method based on binocular image and ground-based radar fusion point cloud

Technical Field

The application relates to the technical field of agricultural detection, in particular to a corn plant height detection method based on binocular image and ground-based radar fusion point cloud.

Background

The accurate agriculture (Precision Agriculture) is a modern agricultural production mode based on information technology and agricultural technology, and the key idea is to accurately monitor and manage information such as crop growth environment, growth condition and yield by utilizing various modern technical means, so that agricultural production benefit and ecological environment protection are improved to the greatest extent. The method mainly aims at determining the production target of crops on the basis of knowing the soil characteristics and the production capacity in the field, performing positioning system diagnosis, optimizing the formula, technical assembly and scientific management to adjust the soil productivity, obtaining equal or higher income with the least investment, improving the environment, efficiently utilizing various agricultural resources and obtaining economic benefit and environmental benefit. The accurate agriculture is a modern agriculture management strategy and agriculture operation technology system based on spatial information management and variation analysis, and realizes accurate field management of positioning and quantification by adjusting crop investment on spatial differences of soil fertility and crop growth conditions so as to realize sustainable development targets of balancing land fertility, improving yield, efficiently utilizing various agricultural resources and improving environment. The implementation of accurate agriculture can not only furthest improve the actual productivity of agriculture, but also be an effective way for realizing high-quality, high-yield, low-consumption and environment-friendly sustainable agriculture development.

Corn is not only one of the main grain crops in the world, but also the grain crop with the largest sowing area and yield in our country, and corn processing byproducts are also important feed sources, and occupy important positions in animal husbandry and aquaculture. In addition, corn is one of the indispensable raw materials in the fields of food, medical and health, light industry, chemical industry, bioenergy and the like, such as corn starch, corn oil, corn wine, corn gluten meal and the like, and is enough to meet the value of corn. The accurate operation of corn comprises accurate sowing, accurate cultivation, accurate fertilization, accurate pesticide application and the like of the corn, so that the purposes of increasing the yield of the corn and fully utilizing the fertilizer are achieved. The method is characterized in that the agricultural planting is guided by monitoring some agricultural traits in the corn growing process, so that the purpose of ensuring the yield and even improving the yield can be achieved by utilizing the accurate agricultural technology.

In crop science research, plant height is an important index, especially for corn. The plant height of the corn is closely related to the lodging resistance of the corn, and is also an important index reflecting the growth cycle and growth vigor of the corn. The plant height of the corn is measured, so that a basis can be provided for field management and corn yield prediction, and the improvement of the nutrition utilization efficiency of the corn is facilitated. From the aspect of nutrition utilization, too short cornstalk can cause insufficient nutrition for corn growth, properly increase the height of corn, be favorable for carbon dioxide to run in the cornstalk, enable the leaf to fully absorb sunlight and promote the growth and development of corn. In addition, the measurement of plant height can also be used for quantitative analysis of genotype and environmental interaction effect, thereby promoting the breeding work of corn. Therefore, research on a method for quickly and accurately obtaining the corn plant height is of great significance for monitoring the corn growth condition.

The traditional corn plant height detection is carried out by manual measurement, and the method needs to enter a field manually for measurement and data recording, and is accurate and reliable, but has low efficiency, lacks real-time performance and can cause certain damage to the field. The method is a urgent need of modern agricultural development in China, and has profound strategic significance for improving the production capacity of modern agriculture in China and ensuring the safety of agricultural production. The development and maturation of autopilot technology and intelligent robotics also make the idea of introducing three-dimensional point cloud technology into agricultural research a reality.

According to the different ways of acquiring the point cloud data, the three-dimensional point cloud technology is divided into two main types, namely active type and passive type. The active type is called an active type because it is required to emit electromagnetic waves, such as laser light or infrared light, to detect a target and then receive reflected waves to acquire depth information of the target. LiDAR (LiDAR) is one of the new representatives of active devices that can generate three-dimensional point cloud data that restores real scenes to high quality. The passive type is to acquire depth information of a target by receiving light emitted from the target or light in the surrounding environment, and is called a passive type because it is not necessary to emit electromagnetic waves, but only to receive light in the surrounding environment or light emitted from the target. Binocular cameras are one of the new representatives of passive devices that use binocular vision techniques to acquire three-dimensional geometric information of an object from an acquired binocular view.

When corn plant height detection is carried out in the prior art, point cloud data which are three-dimensionally reconstructed through binocular images and point cloud data which are acquired through ground radars face the problem that point cloud information at different positions is missing, and the accuracy of detection results is greatly affected.

Disclosure of Invention

The purpose of the invention is that: aiming at the problems that in the prior art, point cloud data which are reconstructed through binocular images in a three-dimensional mode and point cloud data which are acquired through foundation radars face the defect of point cloud information at different positions, and further the precision of single-plant corn plant height detection is reduced, a corn plant height detection method based on the fusion point cloud of the binocular images and the foundation radars is provided

The technical scheme adopted by the invention for solving the technical problems is as follows:

corn plant height detection method based on binocular image and ground-based radar fusion point cloud comprises the following steps:

step one: collecting binocular views of corn plants by using a binocular camera;

step two: calibrating the binocular view of the corn plants by using a Zhang Zhengyou checkerboard calibration method to obtain the internal and external parameters of the camera;

step three: correcting the binocular view of the corn plants by using an polar correction method according to the internal and external parameters of the camera;

Step four: based on the corrected binocular view of the corn plant, combining with an SGBM stereo matching algorithm to obtain a parallax image;

step five: firstly, converting a parallax image into a depth image, and then converting the depth image into point cloud data, namely three-dimensional reconstruction matching point cloud data;

step six: collecting corn plant point cloud data by using a ground-based radar, and denoising the corn plant point cloud data to obtain ground-based radar point cloud data;

step seven: performing target region selection operation on the three-dimensional reconstruction matching point cloud data and the ground-based radar point cloud data;

step eight: and (3) based on the result of the target area selection operation in the step seven, sequentially performing coarse registration and fine registration to obtain corn plant height point cloud data, namely a corn plant height detection result.

Further, the specific steps of the first step are as follows:

firstly, the shooting direction of a binocular camera is in the forward light direction, the binocular camera is arranged on a tripod, then the tripod is moved to enable the binocular camera to be 2-3 m away from corn plants, then the tripod is adjusted to enable the height of the binocular camera to be 10-20cm higher than the top of corn, and a camera of the binocular camera is parallel to the ground for data acquisition.

Further, in the fifth step, the step of converting the disparity map into the depth map is represented as:

depth＝(f*baseline)/disp

wherein depth represents a depth value, i.e., a distance of the camera relative to an object photographed by the camera; f represents the focal length of the camera, i.e. f in the internal parameters of the camera in binocular calibration; baseline represents the baseline distance of the left and right camera optical centers; disp represents the disparity value, namely the disparity value of the left camera and the right camera, which is calculated by an SGBM stereo matching algorithm.

Further, in the fifth step, converting the depth map into point cloud data is represented as:

wherein u and v represent arbitrary coordinate points in the image coordinate system, u ₀ And v ₀ Representing the center coordinate point of the image, x _w ，y _w ，z _w Three-dimensional coordinate points representing world coordinate system, Z _c The Z-axis value representing the camera coordinates corresponds to the distance between the target object and the camera, and dx and dy represent the pixel coordinates in the depth map.

Further, the specific step of acquiring the corn plant point cloud data by using the ground-based radar in the step six is as follows:

firstly, placing a foundation radar on an A-frame, then adjusting the height of the A-frame to enable the foundation radar to be 20 cm-30 cm higher than the top of corn, adjusting the angle of the A-frame to enable the foundation radar to be kept horizontal, then adjusting the foundation radar to panoramic scanning, setting the frame period to 3 seconds, and then placing the foundation radar at a position 2-3 m away from a corn plant to be detected to perform point cloud data scanning.

Further, the coarse registration adopts a 4PCS algorithm, and the fine registration adopts an ICP algorithm.

Further, the target region selection includes automatic segmentation, a deep learning method and manual frame selection.

Further, the target area selection is manual framing.

Further, the binocular camera adopts a ZED2i binocular camera.

Further, the specific steps of calibrating by using the Zhang Zhengyou calibration tool box based on the Matlab platform are as follows:

firstly, obtaining a binocular view of a checkerboard Calibration plate, wherein the checkerboard Calibration plate consists of 12 x 9 black and white alternate lattices, the size of each lattice is 15mm x 15mm, then loading left and right views in the binocular view of the checkerboard Calibration plate into a TOOLBOX after a Matlab platform loads Zhang Zhengyou Calibration TOOLBOX TOOLBOX_calib, starting to extract a focus after clicking an extragrid corner button of the TOOLBOX, and clicking a Calibration button on the TOOLBOX to run a main correction step after completing focus extraction;

and then, carrying out double-target calibration on the two mat files by using a calibration tool box, loading the calibration files of the left and right cameras, and optimizing the intrinsic parameters of the left and right cameras by using a Runstereocoalination function in the tool box, thereby completing the double-target calibration.

The beneficial effects of the application are as follows:

when corn plant height detection is carried out, point cloud data which are three-dimensionally reconstructed through binocular images and point cloud data which are acquired through ground radars face the problem that point cloud information at different positions is missing, and the accuracy of detection results is greatly affected. In order to solve the problem, the application provides a data fusion method, which fuses data acquired by a binocular camera and a ground-based radar on a point cloud layer surface, so that the point cloud information loss of the binocular camera and the ground-based radar at different positions is mutually compensated, and the accuracy of single-plant corn plant height detection is improved.

Drawings

FIG. 1 is a schematic diagram of a binocular camera imaging model;

FIG. 2 is a flow chart of corn plant height detection based on ground based radar;

FIG. 3 is a ZED binocular camera view;

FIG. 4 is a schematic diagram of a binocular camera shooting mode;

FIG. 5 is a schematic diagram 1 of a portion of binocular view data acquired;

FIG. 6 is a schematic view of a portion of binocular view data collected 2;

FIG. 7 is a schematic diagram of a Zhang Zhengyou checkerboard;

FIG. 8 is a schematic view of the left and right eye lens calibration in projection error;

FIG. 9 is a schematic diagram of calibration plate coordinates and pose in camera coordinate system;

FIG. 10 is a SGBM algorithm flow chart;

FIG. 11 is a schematic view of a ground based radar;

FIG. 12 is a schematic view of a ground-based radar acquisition image 1;

FIG. 13 is a schematic view of a ground based radar acquisition image 2;

fig. 14 is a schematic diagram of a coplanar four-point set a, b, c, d in which four long baselines satisfying the requirements are found in the original point cloud set P;

FIG. 15 is a schematic diagram of a co-planar four-point pair finding process;

FIG. 16 is a schematic view of point cloud registration;

FIG. 17 is a flow chart of corn plant height detection for a multisource fusion point cloud;

FIG. 18 is a diagram of ground-based radar point cloud data after target area selection;

FIG. 19 is a graph showing the effect of registration of a binocular image three-dimensional reconstruction point cloud with a ground-based radar point cloud;

fig. 20 is a schematic of a linear fit result.

Detailed Description

It should be noted that, in particular, the various embodiments of the present disclosure may be combined with each other without conflict.

The first embodiment is as follows: referring to fig. 1, a specific description is given of a method for detecting corn plant height based on a fusion point cloud of binocular images and ground-based radars according to the present embodiment, including the following steps:

Binocular stereoscopic vision is a technique that mainly adopts parallax theory to obtain three-dimensional geometric data from a plurality of images. The binocular stereo vision technology generally includes that two digital images of a measured object are obtained by a double camera at the same time under respective visual angles, or two digital images of the measured object are obtained from different visual angles by a single camera in different time, and then three-dimensional geometric data of the object are extracted according to parallax theory, so that the three-dimensional contour and the position of the object are reconstructed. The technique generally includes a series of steps such as image acquisition, image preprocessing, matching algorithms, depth calculation, and point cloud acquisition.

In binocular vision systems, the distance between two cameras is called the baseline, the size of which affects the accuracy and range of depth calculations. In order to obtain the correct depth information, the binocular vision system needs to perform stereo matching, i.e. find the corresponding points in the left and right images, and then calculate the three-dimensional position of the object through the principle of triangulation.

The measurement theory of binocular vision systems is based on the phenomenon of "parallax" that occurs when the human eye observes an object. In computer vision, two cameras can be used for shooting two images at different positions of the same base line, so that the shape and the distance of an object can be judged like human eyes. The binocular vision system may calculate a parallax image of the target image using a positional difference of the target object in two pictures and calculate three-dimensional information of the target object using a similar triangle principle. Fig. 1 illustrates an ideal binocular camera imaging model.

Wherein O is _l And O _r Representing the camera positions of the left camera and the right camera, wherein P is a space point to be solved, and the imaging point of P in the left camera is P ^l The imaging point in the right camera is P ^r Line segment X _l And X _r The distance between the imaging point of the point P in the left and right cameras and the boundary of the imaging surface of each camera is respectively, and the parallax d of the point P in the left and right cameras is expressed as the formula:

d＝|X _l -X _r |

Let P be ^l And P ^r The distance between the two points is s, T is the baseline distance between the left and right two cameras, and the distance s is shown in fig. 1 as the formula:

s＝T-(X _l -X _r )

the formula can be obtained by triangle similarity principle:

wherein f is the focal length of the camera, Z is the distance from the spatial point P to the base lines of the two cameras, and the solution distance Z is the formula:

from the above it can be concluded that no matter which point P in space the corresponding disparity d can be calculated. The magnitude of the disparity d depends on the position of the P point in space, and can be presented by a disparity map after obtaining disparities of all points in space. According to the above formula, as long as the distance T between the two cameras and the camera focal length f are known, the depth map of the image and the three-dimensional coordinates in the world coordinate system can be calculated from the parallax map, and there are various coordinate systems and mapping relations between the two-dimensional imaging map and the three-dimensional world scene. The process of obtaining the parallax map by using binocular vision technology according to the principle is generally binocular calibration, binocular correction and stereo matching.

The binocular calibration is similar to the calibration of a common camera, and internal parameters of the two cameras need to be determined, and meanwhile, the relative position relationship between the two cameras also needs to be determined. Therefore, in the double-object timing, the same calibration board needs to be photographed multiple times, the internal parameters of each camera and the external parameters relative to the calibration board are respectively calibrated, and then the positional relationship between the two cameras can be calculated.

R＝R _r (R _l )T

T＝T _r -RT _l

Wherein R is a rotation matrix between the two cameras, and T is a translation matrix between the two cameras. R is R _r The rotation matrix T of the relative calibration object obtained by the right camera through the Zhongshi calibration is the rotation matrix T of the relative calibration object _r The translation vector of the relative calibration object is obtained for the right camera through the Zhongshi calibration. R is R _l The rotation matrix T of the left camera is obtained by the Soxhlet calibration and is relative to the same calibration object _l And (3) obtaining a translation vector of the left camera relative to the same calibration object through the Zhongshi calibration.

The camera calibration method can be divided into two types, the first is a traditional calibration method requiring a reference object, and the other is a camera self-calibration method without the reference object.

Conventional calibration methods generally use a checkerboard as a reference, where the size, dimension, and number of checkerboards are known. The calibration process is to establish a corresponding relation between the vertex of the checkerboard and the corresponding point on the image, and calculate the internal and external parameters and distortion coefficients of the camera model by using the known information of the checkerboard. The calibration method generally comprises a Zhang Zhengyou calibration method, a Tasi two-step calibration method and the like. This method is susceptible to the accuracy of the manufacture of the calibration object, but the accuracy is still higher than that of the other method.

The camera self-calibration method is a calibration method based on Kruppa equation, and the like, which does not need a reference object. According to a multi-view constraint geometrical equation, images of a plurality of same scenes are acquired at different positions, and calculation of camera parameters is completed through constraint information of a camera and geometrical information of corresponding points. The method has the greatest advantages that a calibration reference does not need to be manufactured, and the method is flexible, but the robustness and the precision are deficient due to the lack of the calibration reference.

After shooting the same object, the two phases subjected to binocular calibration need to be subjected to image correction. The image correction mostly uses epipolar constraint to make the same feature point be positioned on the same straight line in the horizontal direction of the two images of the left and right cameras, namely, "correct the two images which are not in the actual coplanar line alignment into the coplanar line alignment". Of course, some distortion correction will also be performed during this process. After the image correction is carried out by using the epipolar constraint, the characteristic points can be located on the epipolar lines in the two images, so that searching is only carried out on the epipolar lines and not on the whole two-dimensional image when the characteristic points are matched, and the calculated amount is greatly reduced.

When the correspondence between the three-dimensional space and the image is determined, the correspondence between the points of the three-dimensional space on the left and right images must be known to calculate the parallax, which is the purpose of stereo matching. Through the stereo matching technology, the corresponding relation of the points in the left image and the right image can be clarified, so that parallax is obtained, and the three-dimensional information of the points is restored.

Laser radar is also widely used in recent years as a mainstream remote sensing technology for detecting plant heights of crops. The laser radar can accurately measure the distance and the height of an object, has the advantages of high precision, high stability, high definition and the like, and is widely used for measuring and monitoring the plant height of crops. At present, scholars at home and abroad do a lot of researches on a laser radar-based crop plant height detection technology. In terms of hardware, most of the radars used by researchers are high-precision and high-resolution radars, such as Velodyne, riegl, hesai. In the aspect of software, technologies such as machine learning, data mining, computer vision and the like are mainly applied to convert data measured by a laser radar into usable crop height information. The golden person and the like acquire the canopy structure information of the citrus based on the mobile three-dimensional laser radar, and establish a method for acquiring the canopy structure information (tree height, crown width and branch angle) of the citrus. Zhang Pengpeng et al also propose a laser radar-based rice elevation estimation method, which achieves higher accuracy.

The ground-based radar is a laser radar, has the advantages of high precision, high speed, wide view field and the like, and has good transmissivity to the ground covered by vegetation, so that the ground with good effect and the point cloud data of corn plants can be obtained, and the ground-based radar can be used for obtaining the point cloud data of corn farmlands to detect plant height. A corn plant height detection flow chart based on ground-based radar is shown in fig. 2.

The foundation radar is a three-dimensional sensor based on a laser ranging principle, can realize high-precision point cloud acquisition, and is indispensable equipment in three-dimensional reconstruction. The application selects the LivoxMid-100 ground radar as the LivoxMid-100 ground radar. The LivoxMid-100 foundation radar is a lightweight foundation radar which is proposed by Livox (Lipu age) and has the characteristics of high frame rate, high precision, high reliability, low power consumption and the like. The radar has a maximum detection distance of 120 meters, and can provide denser point cloud data by adopting a multi-beam technology. In addition, the LivoxMid-100 foundation radar also has smaller size and weight, can be connected with a computer or other equipment through an Ethernet interface, is easy to integrate into various mobile equipment and robots, and is suitable for the fields of robots, automatic driving, building mapping, environment monitoring and the like.

Binocular camera part

1. Binocular camera and data acquisition

In the data acquisition stage of the binocular camera, the ZED2i binocular camera is used for data acquisition, as shown in FIG. 3, the ZED2i is a ZED binocular stereoscopic camera, and is produced by StereoLabs of French company, and the method is mainly used for depth perception and three-dimensional vision application and has the characteristics of high resolution, high precision, wide visual field and the like.

The shooting direction of the binocular camera is in the forward light direction during data acquisition, so that the influence on the follow-up three-dimensional reconstruction effect caused by unclear shooting targets due to backlight is avoided. When shooting, the binocular camera is arranged on the tripod, the tripod is moved to enable the binocular camera to be 2-3 m away from corn plants, then the tripod is adjusted to enable the binocular camera to be 10-20cm higher than the top of the corn, and the camera is parallel to the ground for data acquisition. Experiments prove that when the distance from the corn plant to be detected is 2-3 m, the depth image picture of the acquired image after three-dimensional reconstruction is clear, the number of invalid points of the image is small, the height of a camera is 10-20cm higher than the top of the corn, and a camera is parallel to the ground, so that the front corn can be prevented from shielding the rear corn, and the problem of incomplete information of the rear corn after three-dimensional reconstruction is avoided. Fig. 4 is a schematic diagram of a binocular camera shooting mode.

The experiment uses a binocular camera to obtain 86 binocular views, the resolution of which is 2560×720 pixels, the format of which is. Jpg, each picture is about 2MB in size, and the binocular views are divided into a left-eye view and a right-eye view in a binocular image. The partial binocular view data acquired by the present application is shown in fig. 5 and 6.

2. Double targeting:

the binocular camera was double-targeted using the Zhang Zhengyou checkerboard method. In the Zhang Zhengyou calibration method, a checkerboard calibration plate is needed for camera calibration. As shown in fig. 7, the self-made checkerboard was composed of 12 x 9 black and white alternate cells, each 15mm x 15mm in size.

The method comprises the steps of calibrating by using a Zhang Zhengyou calibration tool box based on a Matlab platform, firstly splitting and renaming the shot binocular views of 13 groups of checkerboard calibration plates, then loading left and right views shot by a left camera and a right camera into the tool box after loading Zhang Zhengyou the TOOLBOX_calib of the Matlab platform, and starting to extract corner points after clicking Extractgrid cornder buttons of the tool box. Clicking the registration button on the toolbox after focus extraction is completed to run the main correction step. As shown in FIG. 8, the projection errors of the left and right lens are respectively marked and are displayed in a color cross shape.

And then, carrying out double-target calibration on the two mat files by using a calibration tool box, loading the calibration files of the left and right cameras, and optimizing the intrinsic parameters of the left and right cameras by using a Runstereocoalination function in the tool box, thereby completing the double-target calibration. Fig. 9 shows the relative coordinates and the posture of the calibration plate with respect to the camera lens in the experiment, that is, the coordinates and the posture of the calibration plate in the camera coordinate system.

And after Matlab binocular calibration, an internal and external parameter matrix is given. The binocular camera internal reference refers to a projection relation of mapping a camera coordinate system to an image coordinate system, an internal parameter matrix needs to be transposed before use, and translation and rotation parameters of two cameras and an offset matrix of a right camera relative to a left camera can be directly used, but a rotation matrix of the left camera relative to the right camera needs to be transposed before use. Radial distortion (radial distortion) is the distortion that the characteristics of the camera lens cause to be imaged to exist, and is determined by three parameters, K1, K2 and K3. Tangential distortion (taggantialdistorsion) is the imaging distortion caused by errors in the assembly of the optical instrument and sensor, and is determined by both parameters P1 and P2. In summarizing the internal and external parameters, it should be noted that the order of the parameters is K1, K2, P1, P2, K3, otherwise the stereo matching may have a large error. Reading the internal and external parameters of the left and right cameras is very important because they directly affect the final ranging result. The camera outliers reflect the relationship of rotation R and translation T between the camera coordinate system and the world coordinate system. Calibration parameters of the binocular camera are shown in the following table.

3. Binocular camera image correction:

The acquired binocular views are corrected using an epipolar correction method. To achieve this we use the FileStorage function in opencv3.2 to import the calibration camera's internal and external parameters and offset vectors and use the already encapsulated initunderstatorectifimmap function to look up the mapping table. The mapping table maps the original image and the corrected image in a one-to-one correspondence manner, then draws the original image and the corrected image on one image, and draws a straight line at intervals of the same distance, so as to check whether all the points on the object shot by the left camera and the right camera are on the same straight line. If the corresponding matching points of the left and right images have been aligned substantially horizontally and the two pictures have no distortion, as if the left and right eyes of a human were observing the same object, there is no deviation in the corresponding points between them, which means that the two images have undergone correction processing.

4. Binocular camera stereo matching algorithm:

the SGBM stereo matching algorithm is chosen for use. The SGBM (Semi-Global Block Matching) is a Semi-Global stereo Matching (SGM) algorithm, and is a stereo Matching algorithm further optimized based on the SGM algorithm, the main optimization of the SGBM algorithm is in a cost aggregation stage, and the SGBM algorithm uses a left-right cost consistency check and cost aggregation path optimization technology to improve the accuracy and robustness of the algorithm, so that the Matching precision can be improved under the condition of ensuring the efficiency, and particularly in areas with unobvious textures. Compared with the SGM algorithm, the SGBM algorithm has faster calculation speed, because the SGBM algorithm adopts a rapid cost aggregation mode and a minimum value filtering technology is used when searching for cost, the calculation speed of the algorithm can be greatly improved, and the method is suitable for scenes required by real-time processing. And the SGBM algorithm is more convenient than the SGM algorithm and the BM algorithm in parameter adjustment, because the parameter quantity is relatively less, and a more visual parameter adjustment mode is provided, so that the SGBM algorithm is easy to be practically applied. The SGBM algorithm can also accommodate more complex scenes such as varying illumination, different viewing angles, objects of different shapes and textures, etc. In summary, compared with other semi-global stereo matching algorithms, the SGBM algorithm has obvious advantages in the aspects of precision, calculation speed, parameter adjustment, adaptability and the like. The SGBM algorithm flow chart is shown in fig. 10.

5. Binocular camera depth calculation:

converting the disparity map to a depth map is one of the final targets for visual stereo matching. In this process, we need to convert the disparity value (disparity) into a depth value (depth), the unit of the disparity value is a pixel, and the unit of the depth is a millimeter. First, we need to define a concept, namely, the relation between the parallax of the camera and the depth of the object, which can be calculated by the following formula:

depth＝(f*baseline)/disp

in the above formula, depth refers to the calculated depth value, which is the distance between the camera and the object photographed by the camera; f is the focal length of the camera, i.e. f in the internal parameters of the camera in the binocular calibration described above; baseline is the baseline distance between the optical centers of the left and right cameras, and the baseline of the ZED binocular camera adopted by the camera is 12cm; disp refers to a disparity value, and the disparity value of the left camera and the right camera is calculated through an SGBM stereo matching algorithm. And calculating depth values corresponding to all the parallax values according to the formula, and obtaining corresponding depth images.

6. Binocular camera point cloud conversion

Although the depth map is three-dimensional, it is difficult to directly view by naked eyes through image observation or two-dimensional, and when plant height detection is performed, three-dimensional coordinate information of the surface of the corn plant is of great concern, so that the depth map needs to be converted into point cloud data. The point cloud data contains three-dimensional coordinate information of the plant surface, and can more intuitively reflect the shape and structure of the plant, so that the point cloud data can be better used for plant height detection. Specifically, the conversion of the depth map into the point cloud map is a mapping process from world coordinates to a camera coordinate system.

The calculation method of the point cloud data is based on the pinhole imaging principle. The pinhole imaging principle is formulated as:

where u and v are arbitrary coordinate points in the image coordinate system. u (u) ₀ And v ₀ The center coordinates of the images, respectively. X is x _w ，y _w ，z _w Is a three-dimensional coordinate point of the world coordinate system. Z is Z _c Is the Z-axis value of the camera coordinates, corresponding to the distance of the target object from the camera. R is a 3x3 rotation matrix of the outlier matrix and T is a 3x1 translation matrix of the outlier matrix, since the world origin and the camera origin are overlapping and therefore there is no rotation and translation, so R and T are the following matrices, respectively:

the origin of the camera coordinate system overlaps with the origin of the world coordinate system, so the camera coordinates andthe same object in world coordinates will have the same depth, namely Z _c ＝Z _w Equation (1) can then be further reduced to equation (3) as follows:

from the transformation matrix formula, an image point [ u v ] can be obtained] ^T To world coordinate point x _w y _w z _w ] ^T Is a conversion relation of (a).

Foundation radar section

1. Foundation radar and data acquisition

The ground-based radar point cloud data and the binocular view of the binocular camera are simultaneously acquired from a corn planting agricultural area in the open-hearth area of haerbin city of black longjiang province at 6 months and 25 days of 2022. When the ground-based radar is used, data acquisition is performed under the condition of no wind or breeze as much as possible so as to avoid that the wind blows plants and interferes radar signals. Firstly, the foundation radar is arranged on a triangular bracket, the triangular height is adjusted to a position which enables the foundation radar to be 20 cm-30 cm higher than the top of the corn, and the foundation radar is kept horizontal by adjusting the angle. Then, the shooting parameters of the foundation radar are adjusted, the foundation radar is firstly adjusted to a panoramic scanning mode, and the frame period is set to be 3 seconds, so that the acquired data range can be maximized and the acquired data range is most complete. The data frequency of the LivoxMid-100 foundation radar is 300000 points/s, the horizontal view angle is 98.4 degrees, the vertical view angle is 38.4 degrees, and the angle precision is <0.1 degrees. Because the LivoxMid-100 foundation radar cannot scan targets in 1m, the foundation radar is also placed at a distance of 2-3 m from the corn plants to be tested to scan point cloud data. The ground based radar is shown in fig. 11.

The point cloud data files were collected using a ground based radar in 42 format, the las format, with each file size of approximately 22M. Each point cloud file mainly contains information such as position, intensity, classification, time stamp, RGB color, scanning angle and the like of point cloud data. Fig. 12 and 13 are partial effect diagrams of ground-based radar acquisition.

2. Data preprocessing

The point cloud preprocessing mainly comprises two aspects of format conversion and denoising of the point cloud. Format conversion can enable the subsequent use of the PCL library to process the point cloud data more conveniently and rapidly. Denoising can improve the quality of point cloud data, reduce errors, and provide a better data basis for subsequent corn plant height detection.

(1) Format conversion

The point cloud storage data format acquired by the foundation radar is the (las), but the subsequent processing algorithm of the point cloud data is mainly based on a PCL (policy-based computing) point cloud library, and the PCL is a point cloud processing library based on the (pcd) format, so that the point cloud data in the (las) format is converted into the (pcd) format, the PCL library can read, process and visualize the point cloud data more conveniently, and certain modules in the PCL library only support the input data in the (pcd) format, so that the data in the (las) format is converted into the (pcd) format, and the application range of the PCL library can be expanded. Because the pcl_conversion_pcd_ascii_binary tool provided by the PCL library can directly convert the point cloud data in the LAS format into the point cloud data in the PCD format, the application directly completes the format conversion of the point cloud data by using the tool.

(2) Point cloud denoising

When a ground-based radar is used for scanning a corn farmland, some point cloud data can be lost or noise point cloud data which is not intended can be caused due to the influence of shielding of corn blades, stems and the like or the influence of surrounding environments such as light rays, large particulate matters in the air and the like. In addition, ground-based radars themselves may also have some inherent noise, such as the noise of photosensors and electronic components, and the like. Because these noise points can have a great influence on the accuracy of subsequent plant height detection, a point cloud denoising step should be added in the point cloud preprocessing stage.

According to the method, the point cloud data acquired by the foundation radar is denoised in a mode of using a point cloud filtering algorithm (statistical filtering), and because the statistical filtering is a simpler and effective denoising mode, outliers and spurious points in the point cloud can be effectively removed, and the quality of the point cloud data is improved. And the statistical filtering is also convenient to adjust and can be communicatedAnd adjusting the denoising effect by adjusting the number of the neighborhood points and the standard deviation multiple. And when the basic idea of the statistical filtering is that the points around each point in the point cloud are subjected to the statistical analysis and the retrograde motion, judging whether the point is a noise point according to the analysis result, and removing the point judged to be the noise point from the point cloud. The algorithm will find for each point other points within its K neighborhood and then calculate the average distance D from this point to the neighboring points and find the median D of these average distances _med And standard deviation S, if the average distance D between this point and the adjacent point is greater than the maximum distance D _max (D _max ＝D _med +means k x S, where means k is a standard deviation multiple) is removed as noise points. The algorithm can well remove outliers and noise points in the point cloud data, but when the point cloud with the complex shape is processed, parameters are required to be finely adjusted so as to obtain a better filtering effect.

Binocular camera and ground-based radar point cloud fusion part

And selecting and preprocessing a target area before fusing the three-dimensional reconstruction matching point cloud and the foundation radar point cloud. By observing the three-dimensional reconstruction point cloud of the binocular camera and the ground radar point cloud data, the corn point cloud data at a place far away from the acquisition equipment is seriously lost due to the shielding of front corn and cannot be used as a data source for corn plant height detection, so that the corn point cloud data can be divided into noise points and outliers for processing. However, because of the huge and concentrated quantity, the point cloud filtering algorithm cannot be completely removed and is low in efficiency, so that the point cloud data is selected to perform target region selection operation in the preprocessing stage to remove noise points and outliers. The calculation amount of the registration algorithm can be reduced through the target region selection operation, the efficiency of the registration algorithm is improved, and the influence of noise points and outliers on the registration accuracy can be avoided.

The currently commonly used target area selection method comprises an automatic segmentation method, a deep learning method, a manual frame selection method and the like. The automatic segmentation method needs to have a certain priori knowledge on the shape of the target, and has poor adaptability to different scenes; the deep learning method requires a large amount of training data and calculation resources, and has larger limit on practical application; the manual frame selection can flexibly select the target area according to actual conditions to adapt to the requirements of different scenes, and can select the most suitable target area according to the actual conditions observed by human eyes, so that unnecessary interference information is reduced, and the main thing is that the manual wire-framing process is simple and easy to operate and understand, and complex algorithms and technical supports are not needed.

The register AreatingCallback () function provided by the PCL library realizes a manual frame selection function, when the target region selection operation of the corn point cloud data is realized by calling the register AreatingCallback () function provided by the PCL library, the function can register a callback function defined by a user as a response function of a mouse event, and when the user performs the frame selection operation, the function can call the callback function defined by the user and transfer the point cloud data in the frame selection region to the next step for processing. The function is used for selecting the target area, so that the problem of selecting the target area can be solved well, and the function has the advantages of simplicity in operation, wide application range and the like.

1. Binocular three-dimensional reconstruction point cloud and ground-based radar point cloud data registration

The three-dimensional reconstruction matching point cloud and the ground radar point cloud are registered and fused on the point cloud layer surface, so that the complete and accurate point cloud information of the whole corn farmland environment can be obtained, and the accuracy of the subsequent plant height detection of single corn is higher. The cross-source point cloud registration generally adopts a strategy of combining coarse registration and fine registration, and aims to improve registration accuracy and robustness. If the fine registration is directly performed, the position of the point cloud set may be arbitrary, so that the local optimal solution is easily trapped. The coarse registration can provide a better initial position, and then the fine registration is performed, so that the registration process is better completed. In the coarse registration stage, the objective is to find a rough initial registration matrix that brings the position and pose between the point clouds as close as possible. In the fine registration stage, the objective is to minimize the gap between the point clouds by optimizing the initial registration matrix. Therefore, the application selects to use a 4PCS algorithm in a coarse registration stage and uses an ICP algorithm in a fine registration stage to perform registration work of the money point cloud.

The 4PCS algorithm (4-PointsCongrentSets) is a point cloud coarse registration algorithm based on local features. The basic idea of the 4PCS algorithm is to decompose the point cloud characteristics into a set of quaternions, and match the quaternions by calculating the similarity between the quaternions. The 4PCS algorithm is therefore more robust and efficient than other coarse registration algorithms based on local features.

The ICP algorithm is a local fine registration algorithm that can fine align the point cloud that has been coarsely registered. The ICP algorithm transforms one point cloud into the coordinate system of another point cloud by continuous iteration and continuously optimizes the transformation matrix to minimize the error between the two point clouds. Thus, in the fine registration stage, the ICP algorithm is selected to improve the accuracy and robustness of registration.

In summary, the method of combining coarse registration with fine registration can improve the efficiency and precision of point cloud registration, and the 4PCS algorithm and the ICP algorithm can meet the requirements of coarse registration and fine registration respectively.

2. 4 PCS-based point cloud coarse registration

The 4PCS (4-PointsCongrentSets) point cloud coarse registration adopts the concept of quaternion space segmentation, and the point cloud is divided into a plurality of subsets, and each subset only needs to search for the best match, so that the calculation time is greatly reduced. And the 4PCS algorithm can process point cloud registration under the conditions of partial shielding, local deformation and the like, and has certain robustness to outliers and noise in data. The 4PCS algorithm can also be used in combination with other fine registration algorithms to form a complete registration flow, so that the method has better expandability.

The 4PCS (4 poiintscongrouentsets) algorithm is a fast registration algorithm based on point cloud features, and the basic principle is to find a matching point set between two point clouds, so that any four points in the point sets satisfy the affine transformation theorem. The affine transformation theorem in the 4PCS algorithm refers to: on a plane, if two sets of opposite sides of a quadrangle are parallel, respectively, the quadrangle can be changed into a rectangle by affine transformation. Based on the principle, the 4PCS algorithm converts the point cloud matching problem into the problem of finding four points coplanar meeting the affine transformation theorem, and the conversion greatly reduces the search space of matching point pairs, thereby improving the speed and efficiency of registration.

First, four long baseline coplanar four-point sets a, b, c, d satisfying the requirements are found from the original point cloud set P, as shown in fig. 14.

Next, two scale factors for the four coplanar points are calculated according to the following equation. These two scaling factors have affine invariance and therefore their values do not change regardless of whether the point cloud is translated or rotated.

r ₁ ＝||a-e||/||a-c||

r ₂ ＝||c-e||/||c-d||

In another set of point cloud sets to be registered Q, each pair of points in Q is scaled by a scaling factor derived from the point cloud set P Their intermediate points are calculated according to the formula below.

e ₁ ＝q ₁ +r ₁ (q ₂ -q ₁ )

e ₂ ＝q ₁ +r ₂ (q ₂ -q ₁ )

If there are two pairs of such points, one pair is defined by r ₁ Calculated intermediate point and another pair of r ₂ The calculated intermediate points are consistent within the allowed range, then it can be considered that the two pairs of points may be affine corresponding points to the base point in P. The process of finding co-planar four-point pairs is shown in fig. 15.

3. Point cloud fine registration based on ICP

The ICP (IterativeClosestPoint) algorithm is invented by Mckay and Besl rates before one nine two years and two years, and is a registration method for iteration of nearest neighbor points, the main idea is to realize registration of two point clouds by minimizing the distance between the two point clouds, and the problem of fine registration between two three-dimensional point clouds with a certain initial value is solved. The ICP algorithm has relatively little effect on local noise and outliers and thus is robust. Compared with other fine registration algorithms, the ICP algorithm has the advantages of high precision, strong robustness, simple algorithm and the like, and is widely applied to the problem of multi-point cloud registration.

As shown in fig. 16, which is a flow chart of the ICP algorithm, the ICP algorithm host needs to perform two key steps: find the nearest corresponding point and calculate the optimal transformation parameters. Assume that there is a set S of point clouds to be registered _p And target point cloud set T _q Firstly, finding out a point p with the minimum distance according to Euclidean distance between two point cloud sets _i And q _i Regarding the two adjacent points as the nearest points, calculating corresponding translation vectors T and rotation matrixes R from the found nearest point pairs, and finally bringing the translation vectors T and the rotation matrixes R into the following formula to obtain an error function E (R, T), wherein the error function is defined as follows: where n is the total number of nearest neighbors.

And then carrying out coordinate conversion on the point cloud to be registered according to the obtained translation vector T and the rotation matrix R, and then calculating the nearest point between the two point cloud sets and the corresponding translation vector T and rotation matrix R again according to the calculation flow until the calculated error function is lower than a set threshold value or the iteration number exceeds a set maximum value, and ending the algorithm.

If the number of the corresponding nearest neighbor points is insufficient when calculating the optimal transformation parameters, a SVD (singular value decomposition) method can be used for solving the translation vector T and the rotation matrix R. The translation vector T and rotation matrix R can be solved using a nonlinear least squares method if the corresponding number of nearest neighbor points is large. The algorithm flow of SVD includes: calculating the mass centers of the two point clouds, calculating a centralization vector, calculating a covariance matrix and singular value decomposition, and solving a rotation matrix R and calculating a translation vector T. The detailed process is as follows:

The above known point cloud registration is formed by solving a rotation matrix R and a translation vector T, and the objective function is recorded as follows:

in the above formula, n is the number of matching points, and the least squares solution is set to R 'and T', then q _i ′＝R′p _i The +T' and Q centroids are the same, i.eWherein:

and the centroid of the point cloud PThe method comprises the following steps:

then, the following steps:

the objective function can then be rewritten as:

by decomposing and transforming J into:

to minimize J, one needs to derive J and maximize J', then there are:

in the above description, H is the third-order policy:

the decomposition of the H matrix with SVD has:

H＝U∧V ^T

let x=vu ^T The following steps are:

XH＝VU ^T U∧V ^T ＝V∧V ^T it can be seen that XH is a symmetric positive definite matrix. Therefore, if tr (XH) > tr (BXH) is given to any of the third-order orthogonal square matrices B, the rotation matrix r=x is only given that the determinant of X is close to 1 or equal to 1 (complete coincidence after transformation) among all the third-order orthogonal square matrices. Then the translation matrix is:

4. fusion process

According to the algorithm, the corn plant height is detected after the three-dimensional reconstruction matching point cloud and the foundation radar point cloud are fused, and the detection result is analyzed and compared. The experimental input data are three-dimensional reconstruction matching point cloud data and ground radar point cloud data subjected to format conversion and denoising. Fig. 17 is a flowchart of corn plant height detection based on binocular image three-dimensional reconstruction point cloud and ground-based radar point cloud data fusion.

In the target region selection stage, the application is realized through the manual framing function of the register image PickingCallback () function provided by the PCL library, and a user can perform framing cutting on the part with complete space information of the point cloud data according to own observation, so that the influence on the efficiency and the robustness of subsequent registration and plant height detection is avoided. Fig. 18 shows the ground-based radar point cloud data after the target area selection.

In the point cloud registration stage, the application uses a point cloud registration strategy combining 4PCS coarse registration and ICP fine registration to perform registration fusion on the binocular image three-dimensional reconstruction point cloud and the ground radar point cloud data, and the registration effect of the binocular image three-dimensional reconstruction point cloud and the ground radar point cloud is shown in fig. 19.

In order to be distinguished easily, the method selects the ground-based radar point cloud to be marked green, the three-dimensional reconstruction matching point cloud to be marked white, and then the registration accuracy is analyzed from two aspects of qualitative and quantitative.

First, quantitative analysis can be performed by calculating the Root Mean Square Error (RMSE) of the matched pairs of points. RMSE (Root Mean Square Error) is an indicator of the effectiveness of point cloud registration and is typically used to compare the difference between two point clouds. The specific calculation method of the RMSE comprises the following steps: after the point cloud registration is completed, searching the nearest point in the source point cloud for each target point, calculating the square of the distance between the two corresponding points, and obtaining the sum of the squares of the distances of all the corresponding points, wherein the ratio of the sum to the square root of the number of the corresponding points is root mean square error. The smaller the root mean square error, the better the match between the two point clouds. The formula is as follows:

And finally, the root mean square error is calculated to be 2.21cm, so that the registration accuracy is higher, and the error is in an acceptable range.

From qualitative angle analysis, by observing the superposition display effect graph of the three-dimensional reconstruction matching point cloud and the ground radar point cloud, the two point clouds can be obviously overlapped greatly after registration and fusion, the degree of fit is higher, and the point clouds at different positions are mutually complemented. The three-dimensional reconstruction point cloud data of the binocular image complements the point cloud information of the ground-based radar on the corn canopy, the point cloud information of the three-dimensional reconstruction matching point cloud on the corn lower layer and the ground is complemented by the ground-based radar point cloud, so that the advantages of the multi-source point cloud fusion data in the aspects of whole and details are verified, and the precision of the point cloud fusion can be further evaluated through the detection result of the plant height of the single corn.

5. Experimental results

And separating ground points by using a cloth filtering algorithm for the fused point cloud, then detecting the corn plant height of the multisource fused point cloud data according to a plant height detection method based on a ground radar, analyzing a system measurement result, and analyzing the plant height detection result by selecting 21 samples the same as the two-dimensional camera and the ground radar for precision comparison with a method for detecting the corn plant height by using the two-dimensional camera and the ground radar. The corresponding system measurements and actual measurements are shown in the table below.

A linear fit of the measured values to the total 52 sets of system measurements is shown in FIG. 20, where the coefficient R is determined ² =0.9814. In the corn plant height estimation result of spot check, the average measurement error of the corn plant height is 0.99cm, the RMSE is 1.17cm, the error rate of the average plant height is 1.85%, the maximum error of the individual corn plant height is 7.54%, and the minimum error of the individual corn plant height is 0.82%.

As can be seen from analysis of the corn plant height detection results of the multisource fusion point cloud, the corn plant height error rate is generally lower than that of corn plant height detection using a binocular camera and ground based radar alone. In addition, the stability of the multisource fusion point cloud is far higher than that of a mode of plant height detection by using a binocular camera. The advantages of the multi-source point cloud fusion data in the aspects of both the whole and the details can be further proved through the results, and the effect of improving the detection precision of the multi-source fusion point cloud on the corn plant height is also verified.

It should be noted that the detailed description is merely for explaining and describing the technical solution of the present invention, and the scope of protection of the claims should not be limited thereto. All changes which come within the meaning and range of equivalency of the claims and the specification are to be embraced within their scope.

Claims

1. The corn plant height detection method based on binocular image and ground-based radar fusion point cloud is characterized by comprising the following steps:

2. The method for detecting the corn plant height based on the fusion point cloud of the binocular image and the ground-based radar according to claim 1, wherein the specific steps of the first step are as follows:

3. The method for detecting corn plant height based on binocular image and ground-based radar fusion point cloud according to claim 2, wherein in the fifth step, the parallax map is converted into a depth map, which is represented as:

depth＝(f*baseline)/disp

4. The method for detecting corn plant height based on binocular image and ground-based radar fusion point cloud according to claim 3, wherein in the fifth step, the depth map is converted into point cloud data, which is represented as:

5. The method for detecting the corn plant height based on the fusion point cloud of the binocular image and the ground-based radar according to claim 4, wherein the specific step of acquiring the corn plant point cloud data by using the ground-based radar in the step six is as follows:

6. The corn plant height detection method based on the binocular image and ground-based radar fusion point cloud, which is characterized in that the coarse registration adopts a 4PCS algorithm, and the fine registration adopts an ICP algorithm.

7. The method for detecting the corn plant height based on the fusion point cloud of the binocular image and the ground-based radar according to claim 6, wherein the target area selection comprises an automatic segmentation method, a deep learning method and a manual frame selection.

8. The method for detecting corn plant height based on binocular image and ground based radar fusion point cloud as claimed in claim 7, wherein the target area is selected as a manual frame selection.

9. The corn plant height detection method based on the fusion point cloud of the binocular image and the ground-based radar according to claim 1, wherein the binocular camera is a ZED 2i binocular camera.

10. The method for detecting the corn plant height based on the fusion point cloud of the binocular image and the foundation radar according to claim 1, wherein the specific steps of calibrating by using a Zhang Zhengyou calibration kit based on a Matlab platform are as follows:

firstly, obtaining a binocular view of a checkerboard Calibration plate, wherein the checkerboard Calibration plate consists of 12 x 9 black and white alternate lattices, the size of each lattice is 15mm x 15mm, then loading left and right views in the binocular view of the checkerboard Calibration plate into a TOOLBOX after a Matlab platform loads Zhang Zhengyou Calibration TOOLBOX TOOLBOX_calib, starting to extract a focus after clicking a Extract gridcornder button of the TOOLBOX, and clicking a Calibration button on the TOOLBOX to run a main correction step after completing focus extraction;

and then, performing double-target calibration on the two-mat files by using a calibration tool box, and optimizing the intrinsic parameters of the left and right cameras by loading the calibration files of the left and right cameras and using Run stereo calibration functions in the tool box, thereby completing double-target calibration.