CN115880500A - Tractor detection method, apparatus, system, device, medium, and program product - Google Patents
Tractor detection method, apparatus, system, device, medium, and program product Download PDFInfo
- Publication number
- CN115880500A CN115880500A CN202210134625.1A CN202210134625A CN115880500A CN 115880500 A CN115880500 A CN 115880500A CN 202210134625 A CN202210134625 A CN 202210134625A CN 115880500 A CN115880500 A CN 115880500A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- data
- tractor
- target
- training data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a tractor detection method, equipment, a system, a device, a medium and a program product, wherein the method comprises the following steps: acquiring point cloud data of a tractor in an agricultural scene; obtaining a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result; the target tractor detection model is obtained through the following steps: acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene; performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data; constructing a target point cloud data set with labels based on the point cloud training data and the image training data after time synchronization and space synchronization; and inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model. The tractor detection device is used for solving the defect that the detection accuracy rate of a plurality of tractors is reduced because only a specific tractor can be detected.
Description
Technical Field
The invention relates to the technical field of agriculture, in particular to a tractor detection method, equipment, a system, a device, a medium and a program product.
Background
At present, detection research on tractors aiming at common working scenes (hangars, machine plowing roads and farmlands) of running and operation of agricultural machines mainly focuses on images or laser point cloud data. The tractor detection based on the image is mature in the aspects of the traditional method and deep learning, and the tractor detection method based on the laser point cloud mainly utilizes a clustering method in the traditional method, but the deep learning method is less in research.
The tractor detection method based on the laser point cloud mainly extracts, classifies and detects the features of the tractor on the basis of a clustering method in the traditional method, but the method is easy to cause low generalization, only can detect a specific tractor and reduces the accuracy rate of detecting a plurality of tractors.
Disclosure of Invention
The invention provides a tractor detection method, equipment, a system, a device, a medium and a program product, which are used for solving the defects that only a specific tractor can be detected and the accuracy rate of detecting various tractors is reduced in the prior art.
The invention provides a tractor detection method, which comprises the following steps:
acquiring point cloud data of a tractor in an agricultural scene;
obtaining a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result;
wherein the target tractor detection model is obtained by the following steps:
acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene;
performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data;
constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization;
and inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
According to the tractor detection method provided by the invention, the constructing of the labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization comprises the following steps:
projecting point cloud training data onto a target image, and simultaneously marking the target image and the point cloud training data in a visual field range corresponding to the target image by using a competition semi-automatic marking tool; the target image is image training data that is time-synchronized and space-synchronized with the point cloud training data;
setting labels for the labeling results in a preset label form to obtain a target point cloud data set with the labels; the preset label form at least comprises a two-dimensional boundary frame, a three-dimensional frame size, a three-dimensional frame position and an observation angle of the tractor.
According to the tractor detection method provided by the invention, before the step of constructing the cloud data set of labeled target points based on the point cloud training data and the image training data after time synchronization and space synchronization, the method further comprises the following steps of:
and carrying out distortion correction on the point cloud training data based on the vehicle pose data.
According to the tractor detection method provided by the invention, the distortion correction of the point cloud training data based on the vehicle pose data comprises the following steps:
acquiring pose information which is correspondingly matched with each laser point in each frame of point cloud acquired by the laser radar based on the vehicle pose data;
and converting the coordinates of each laser point in each frame of point cloud into the coordinates of the first laser point in each frame of point cloud under a coordinate system based on the pose information.
According to the tractor detection method provided by the invention, before the step of inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain the target tractor detection model, the method further comprises the following steps:
extracting a target area with a preset three-dimensional size from each frame of point cloud data of the target point cloud data set;
and filtering non-ground point cloud data in the target area by using a ground point cloud denoising method based on a plane model.
The invention also provides a tractor detection device, comprising:
the point cloud data acquisition module is used for acquiring point cloud data of the tractor in an agricultural scene;
the detection result acquisition module is used for acquiring the detection result of the tractor: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result;
wherein the target tractor detection model is obtained by:
the data acquisition module is used for acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene;
the synchronization module is used for carrying out time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data;
the data set construction module is used for constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization;
and the training module is used for inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the tractor detection method as described in any one of the above.
The invention also provides a tractor detection system which comprises a laser radar for acquiring point cloud training data, a camera for acquiring image training data, a combined navigation device for acquiring vehicle pose data and the electronic equipment.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the tractor detection method as described in any one of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, carries out the steps of the tractor detection method as described in any one of the above.
According to the tractor detection method, the device, the equipment, the storage medium and the program product, the multi-source training data consisting of the point cloud training data, the image training data and the vehicle pose data of various tractors in the agricultural scene are acquired, the multi-source training data are subjected to time synchronization and space synchronization, and the target point cloud data set with the labels is constructed based on the image training data and the point cloud training data. And inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model. And finally, detecting the tractor based on the target tractor detection model. The multi-source sensor data of various tractors in the agricultural scene are collected, and the target point cloud data set with the labels and with a large amount of point cloud data is constructed based on the two-dimensional image training data and the three-dimensional point cloud training data of various tractors in the agricultural scene, so that the target point cloud data set with the labels has more kinds of tractor characteristic information, a target tractor detection model obtained by training the target point cloud data set with the labels is suitable for detecting tractors in wider agricultural scenes, and the accuracy of detecting various tractors is improved. The defect that the accuracy rate of detecting a plurality of tractors is reduced due to the fact that only specific tractors can be detected currently is overcome.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following briefly introduces the drawings needed for the embodiments or the prior art descriptions, and obviously, the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic flow diagram of a tractor inspection method according to the present invention;
FIG. 2 is a schematic flow chart of the spatial synchronization of vehicle pose data and point cloud training data according to the present invention;
FIG. 3a is a schematic diagram of a feature obtained after point cloud projection at a geodetic coordinate position;
FIG. 3b is a schematic diagram of the feature position in the earth coordinate obtained after the point cloud projection;
FIG. 4 is a schematic of the time synchronization of point cloud training data, image training data, and vehicle pose data of the present invention;
FIG. 5 is a schematic view of the viewing angle of the present invention;
FIG. 6 is a second schematic flow chart of the tractor detection method provided by the present invention;
FIG. 7a is a schematic diagram of a laser point collected in a frame of point cloud when the laser radar has no pose change;
FIG. 7b is a schematic diagram of the laser spot collected in a frame of point cloud when the laser radar has a pose change;
FIG. 8 is a schematic flow chart of distortion correction of point cloud training data based on the vehicle pose data according to the present invention;
FIG. 9 is a second schematic flow chart of the tractor detection method provided by the present invention;
FIG. 10a is a side view of a target area of a predetermined three-dimensional size in an extracted target point cloud dataset according to the present invention;
FIG. 10b is a top view of a target area of a predetermined three-dimensional size in the extracted target point cloud dataset according to the present invention;
FIG. 10c is a perspective view of a target area of a predetermined three-dimensional size in the extracted target point cloud dataset according to the present invention;
FIG. 11 is a schematic structural view of a tractor testing apparatus provided by the present invention;
FIG. 12 is a schematic structural diagram of an electronic device provided by the present invention;
fig. 13 is a schematic structural view of a tractor detection system provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
The tractor detection method of the present invention will be described with reference to fig. 1-9.
Referring to fig. 1, the tractor detection method provided by the invention comprises the following steps:
and 200, acquiring point cloud data of the tractor in the agricultural scene.
The tractor can be various tractors used in common working scenes (such as hangars, tractor plowing roads and farmlands) of running and operation of various agricultural machines. The point cloud data is a three-dimensional image formed by 3D imaging of the tractor. Specifically, the laser radar can be used as a three-dimensional perception sensor of the tractor, so that the point cloud data of the tractor can be acquired. The point cloud data refers to data which is specially used for inputting a target tractor detection model to detect the tractor.
Step 300, obtaining a tractor detection result: and inputting the point cloud data into a target tractor detection model to obtain a tractor detection result.
Wherein the target tractor detection model is obtained by the following steps:
Specifically, the point cloud training data and the image training data refer to training data for training to obtain a target tractor detection model. The point cloud training data can be used as a three-dimensional perception sensor of the tractor through the laser radar, and therefore point cloud data of the tractor can be collected. The image training data may be collected by a camera.
Specifically, the laser radar can be 16-line laser radar VLP-16 of Velodyne. The horizontal angle of view of the laser radar VLP-16 is 360 degrees, the vertical angle of view can reach 30 degrees (+/-15 degrees), the laser beam is 16 layers, the outer shape is small, the installation mode is simple, the laser radar VLP-16 is suitable for three-dimensional sensing in a low-speed environment, and the laser radar VLP-16 can be installed on a tractor. The average speeds of the tractor on the tractor plowing channel and the farmland are respectively 28km/h and 10km/h, and 288000 data points can be obtained every second when the acquisition frequency is 10HZ, so that the acquisition requirement of the tractor point cloud is met.
The camera for acquiring the image training data can adopt an industrial camera which is manufactured by Shanghai Saint Emerson Vision technology, inc. and is of the model FLIR BFS-PGE-23S3, the resolution of the camera is 1920X1200 (230 ten thousand pixels), the maximum compression frame rate can reach 80 frames per second, the frame rate under the full resolution is 53 frames per second, and the acquisition modes of single frames and continuous multiple frames are supported.
The vehicle pose data refers to data for pose determination for the tractor. Specifically, the vehicle pose data can be measured by a Hua-measuring CGI-610 centimeter-level integrated navigation device installed on a tractor. The positioning directional antenna is installed on the roof of the tractor. The combined navigation device combines satellite positioning and inertial measurement, adopts a high-precision positioning and orientation GNSS technology, supports a 555 channel, adopts a 2.5-degree zero-offset high-precision gyroscope and an accelerometer, supports external odometer information to perform auxiliary calculation, improves reliability and dynamic property by means of a multi-sensor fusion technology, and can provide high-precision carrier position, attitude, speed and sensor information in real time. The application requirements of long-time, high-precision and high-reliability navigation in complex environments such as agricultural machinery operation environment and the like are well met.
And 120, performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data.
And the electronic equipment carries out time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle posture data.
First, since the information collected by each sensor is based on its own coordinate system, for example, the point cloud training data is point cloud data obtained under a laser radar coordinate system, and the image training data is image data obtained under a camera coordinate system. The point cloud training data, the image training data and the vehicle pose data of different coordinate systems need to be spatially synchronized.
Specifically, the time synchronization and the space synchronization are performed on the point cloud training data, the image training data and the vehicle pose data, and the method comprises the following steps:
and step 121, converting the point cloud training data in the laser radar coordinate system into image training data in the camera coordinate system based on the internal reference and the external reference of the camera.
Specifically, a checkerboard calibration board can be used as a reference, the laser radar scans the checkerboard to acquire position information of a certain point under the definition of the laser radar, the coordinate is (X, Y, Z), meanwhile, the camera scans the checkerboard to acquire the coordinate (u, v) of the point on a pixel plane, the point (X, Y, Z) in the laser radar is calibrated through external parameters after depth information is removed, and then the internal reference of the camera can be used for converting to (u, v) under a pixel coordinate system.
Assuming the coordinate system of the lidar: o is l -X l Y l Z l (ii) a The coordinate system of the camera is as follows: o is c -X c Y c Z c (ii) a The coordinate system of the calibration plate is as follows: o is b -X b Y b Z b Converting the point cloud data from the laser radar coordinate to the camera coordinate system according to the following formula:
the formula for transforming the camera coordinate system to the pixel plane is as follows:
wherein R is a rotation matrix between the laser radar and the camera, and T is a translation vector between the laser radar and the camera. K is camera reference, and the normal vector P under the camera coordinate system is shown in the following formula ci Conversion to laser radar coordinate system
Normal vector P in coordinate system of laser radar li And therefore, a normal vector matrix formed by the laser radar and the corresponding feature points in the camera coordinate system is shown in the following formula (3).
P c ＝[P c1 ,P c2 ...P cn ]
P l ＝[P l1 ,P l2 ...P ln ](ii) a Formula (3)
Wherein the rotation matrix R has the following properties:
due to the fact that
And P li In parallel, R can be determined so that->
And P li The sum of the cosine of the angle is the largest as shown in the following formula (5)
Finally, R = VU can be obtained by resolving R T Analytic solution of translation vector T is the same as
And step 122, carrying out spatial synchronization on the vehicle pose data and the image training data.
The camera is easy to lose efficacy under the conditions of high-speed movement and illumination change, the combined navigation device can acquire the movement information of the vehicle, is not influenced by the surrounding environment and can make up for the defects of the camera, so that the vehicle pose data and the image training data can be subjected to spatial synchronization.
Specifically, the internal reference calibration of the camera is firstly carried out, and the method comprises four steps:
(1) Acquiring a calibration plate corner point from the image;
(2) For the initial value, points in each line of the calibration plate can be fitted into a circle after imaging projection according to the initial value, the two circles are intersected at two points, and the focal length is the distance between the two points divided by pi; the image size is typically the principal point coordinate (c) x ,c y )；
(3) The calibration plate is a fixed reference coordinate system, the size of the calibration plate is known, the three-dimensional coordinates of the angular points under the reference system of the calibration plate can be obtained, and the three-dimensional coordinates correspond to the two-dimensional coordinates, so that the pose of each frame of the camera can be obtained;
(4) And converting the three-dimensional points under the coordinate system of the calibration plate into the coordinate system of the camera by utilizing the pose of each frame, and predicting the projection position of the three-dimensional points to the image through a camera projection model. And continuously optimizing the pose and the internal parameters of each frame of camera so as to minimize the reprojection error.
The internal reference of the camera can be calibrated through the steps.
And then carrying out external reference calibration on the camera and inertial navigation. Acceleration and angular velocity at each time can be acquired from the inertial sensor information of the integrated navigation device, velocity, position, and rotation information can be acquired by integrating the acceleration and angular velocity, and the discrete points are connected in series by a bezier curve to inversely synthesize B-spline.
(1) A time delay between the camera and inertial navigation is estimated. The pose of each frame of image can be obtained by the camera internal reference calibration, the discrete poses are inversely synthesized into B-spline by utilizing a Bezier curve, the rotation angular velocity of the camera at any moment can be obtained, the angular velocity of the combined navigation can be obtained from the gyroscope, the influence of offset noise is neglected, and the following formula (6) can be obtained:
w i ＝R ic w c (ii) a Formula (6)
Wherein w i Angular velocity of rotation, w, of imu c Is the angular velocity of rotation, R, of the camera ic Is the imu to camera rotation matrix. Estimating the time delay of the camera and the combined navigation according to the correlation of the two B-profiles, as shown in the following formula v i+k The velocity at time i + k.
(2) Acquiring an initial value: rotation matrix between inertial navigation and camera, gravitational acceleration, gyroscope bias. Using the angular velocity measurement relationship of step 2, an optimum relationship is constructed as shown in the following equation (8), b g Is the initial value of the gyroscope bias.
R ic ,b g ＝arg max∑R ic w c +b g -w m (ii) a Formula (8)
Equation (9) of the initial value of the gravitational acceleration in the reference system is as follows:
(3) And optimizing an inertial sensor (IMU) accelerometer, a gyroscope measuring error and a corner re-projection error according to the initial values obtained in the first two steps.
The external parameters of the camera can be calibrated through the steps.
And step 123, carrying out spatial synchronization on the vehicle pose data and the point cloud training data.
Calibrating the laser radar and the combined navigation device: the target objects in the same scene are generally fixed and unchangeable, if the external parameter coefficients of the laser radar are relatively accurate, the point clouds of different frames are projected to a geodetic coordinate system in combination with the position information of combined navigation in different directions, and good contact ratio is required, if the initial external parameters of the laser radar are utilized, iteration is carried out in the direction of increasing the contact ratio, and then the corrected external parameter information of the laser radar can be calculated.
Based on the above basis, referring to fig. 2, the main steps of the present invention for performing spatial synchronization on the vehicle pose data and the point cloud training data are as follows:
(1) Position measurement: measuring the displacement of the laser radar relative to the combined navigation device on (X, Y, Z) by using a measuring tool, wherein the error is within the range of 5 cm; under the conditions that the laser radar is horizontally placed and the data derivation line faces the tail of the vehicle, the Roll angle (Roll) and the horizontal angle (Pitch) are 0 degree, and the Yaw angle (Yaw) is 90 degrees; in other installation conditions, the angle value is required to be measured, the error is within 4 degrees, and the transformation matrix T from the laser radar to the integrated navigation device is obtained by utilizing the relative displacement (X, Y, Z, roll, pitch, yaw) 0 ；
(2) Data acquisition: recording point cloud data and geodetic coordinate data of a combined navigation device in a scene with obvious fixed features, and randomly extracting 100 frames of point cloud data (P) in a data packet 0 ,P 1 ...P k ) And corresponding geodetic coordinate data (W) of the combined navigation 1 ,W 2 ...W k ) Converting the geodetic coordinate data into a conversion matrix from the integrated navigation device to the geodetic coordinate system
Storing the data;
(3) Point cloud projection: point cloud data (P) 0 ,P 1 ...P k ) Sequentially pre-multiplying each frame point cloud by a transformation matrix T for the coordinate position of the peripheral characteristic object relative to the laser radar 0 Obtaining the relative characteristicsCombining the coordinate positions of the navigation device, and then carrying out left multiplication on the IMU-group conversion matrix T corresponding to each frame w And obtaining the coordinate position of the characteristic object relative to the earth, wherein the earth coordinate system is unique, and therefore, the actual position of the characteristic object is finally obtained. The method can obtain the projected point cloud data
(4) Iterative optimization: conversion matrix T due to IMU-group w Higher precision if the matrix T is transformed 0 Inaccurate, feature position in earth coordinate obtained after point cloud projection
The degree of mutual overlap is low, and double images occur among the features, as shown in fig. 3 (a), the double images exist among the feature 1 in the three-dimensional environment, the feature 2 projected at the a position, and the feature 1 projected at the B position; if the matrix T is transformed 0 Accurate, feature obtained from point cloud projection is located on the geodetic coordinate>
As shown in fig. 3 (B), since there is a small ghost among the feature 1 in the three-dimensional environment, the feature 2 projected at the a position, and the feature 1 projected at the B position, a multi-step iteration method is sequentially used for six parameters in the displacement (X, Y, Z, roll, pitch, yaw) of the laser radar with respect to the integrated navigation device, and data ∑ is obtained by calculation based on T>
The coefficient of overlap ratio M between them, the value M of the minimum coefficient of statistical overlap ratio k Will be in contact with M k The corresponding parameter combination is used as the iteration center of the next step to carry out the next iteration, and the six parameters are iterated in sequence to obtain the finally optimized T best 。
In the data acquisition process, because the laser radar and the camera are independently packaged and have own time periods, the laser radar and the camera independently operate according to the own time period reference, the acquisition frequencies of all the sensors are inconsistent, and data transmission among different sensors has certain delay, the sensors cannot be ensured to acquire synchronous information at the same time. The embodiment can realize time synchronization between the point cloud training data and the image training data and time synchronization between the vehicle pose data and the point cloud training data through soft synchronization or hard synchronization.
And step 124, performing soft synchronization or hard synchronization on the point cloud training data and the image training data.
Specifically, if the soft synchronization is used to realize the time synchronization between the point cloud training data and the image training data, different topics needing to be fused with the sensors can be respectively subscribed through the ROS, and the topics are uniformly received and stored in the message filter, and only when topics with the same timestamp appear, information corresponding to the topics is output. Thereby obtaining point cloud training data and the image training data which are synchronized in time.
Referring to fig. 4, if time synchronization between the point cloud training data and the image training data is implemented by using hard synchronization, there are three lines in the laser radar with PPS pulse signals output, one of which is connected to PIN2 of the camera, and the camera is triggered when the PPS is at a rising edge. The laser radar continuously scans, 360 degrees are frame point cloud data, the point cloud framing angle is used as a starting angle and an ending angle, and the point cloud framing angle can be arranged outside the camera view field. Thus, the frequency of triggering the camera may be determined by adjusting the point cloud framing angle. This writing sets up the point cloud framing angle to 180 degrees, and with the phase locking angle 0 degree, guarantees the synchronization of image training data and point cloud training data, can make laser radar trigger the camera with 20Hz frequency simultaneously, obtains more camera images, follows up with laser radar time as the benchmark time, aligns the time stamp through the time soft synchronization.
And step 125, performing soft synchronization or hard synchronization on the point cloud training data and the vehicle pose data.
For soft synchronization, the ROS can also subscribe to different topics needing sensor fusion respectively, and receive the topics uniformly, the topics are stored in the message filter, and only when topics with the same timestamp appear, the information corresponding to the topics is output. And obtaining time-synchronized point cloud training data and the vehicle pose data.
For hard synchronization, referring to fig. 4, the GNSS receiver in the integrated navigation device outputs GPRMC data with precise time stamp information and PPS synchronization pulse signals, as shown in fig. 4, which shows the GNSS synchronization principle, the GPRMC information is cleared 300ms before the next PPS signal arrives, the PPS synchronization pulse length is 20us to 200ms, and the GPRMC data must be completed within 500ms of the synchronization pulse. The PPS pulse signals and the UTC (universal time control) second point are accurately aligned, the frequency of the PPS pulse signals is 1Hz, the frequency of other data output can be higher than 1Hz, the PPS pulses are connected with a GPS _ REC interface of the laser radar, the laser radar can rotate a specific angle to emit laser when the rising edge of the PPS pulses is triggered, and the sensor respectively rotates to 0 degree, 135 degrees and 270 degrees to emit laser. The sensor can completely align the time stamp between the two at the initial time, the clock precision is timed by the sensor before the next pulse signal arrives, and at the moment, the precise time of the laser radar data packet is the precise UTC time obtained from the GNSS receiver and the value in the counter. The precision of the counter is related to a clock used by the sensor, the clock error is assumed to be 2ppm, the maximum error of the clock within 1s is 2us, when the rising edge of the PPS signal is triggered next time, the counter is cleared, the whole-second timer is added by 1, and on the premise of not considering the error of the PPS pulse signal, the maximum error of the laser radar timestamp does not exceed 2us no matter how long the time is. The precise time of each laser radar data packet can be obtained through recursion, and the time at the moment is consistent with the time of the GPS output data.
And 140, constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization.
And the electronic equipment constructs a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization.
Specifically, the constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization includes:
step 141, projecting the point cloud training data onto a target image, and simultaneously labeling the target image and the point cloud training data in the corresponding visual field range of the target image by using a double match semi-automatic labeling tool; the target image is image training data that is time-synchronized and space-synchronized with the point cloud training data;
due to the sparsity and abstraction of the point cloud data, images corresponding to the point cloud data in time are needed, and due to the fact that the view angle range of the laser radar is 360 degrees, the view angle range of the camera is far smaller than that of the laser radar, the point cloud data need to be projected onto the images for more accurate marking.
The invention uses a double-race semi-automatic marking tool to extract external parameters in the marking results of the camera and the laser radar, so that point cloud training data are projected into the camera, the marking range of the point cloud is determined, and only the point cloud in the visual field range of the camera needs to be marked during marking.
According to the method, the three-dimensional point cloud training data are accurately marked by combining the two-dimensional image information of the image training data, so that the data quality of the obtained target point cloud data set is improved, and the detection accuracy of the obtained target tractor detection model is improved.
142, setting labels for the labeling results in a preset label form to obtain a labeled target point cloud data set; the preset label form at least comprises a two-dimensional boundary frame, a three-dimensional frame size, a three-dimensional frame position and an observation angle of the tractor.
And under the condition of marking, setting a label at the marking position in a preset label mode to obtain a target point cloud data set with the label. The preset label-shaped final result gives coordinates of a center point of a three-dimensional frame of the tractor (namely, a position of the three-dimensional frame), a length, a width, a height and a rotation angle yaw of the three-dimensional frame (namely, a size of the three-dimensional frame), and coordinates of an upper left corner and a lower right corner of the two-dimensional frame in a json format.
Specifically, in some embodiments, the present invention sets the preset label form in the whole scene to the format of the following table 1 according to the data label format of KITTI (jointly launched by karsleu physical institute of germany and technical research institute of toyota, which is the currently internationally largest computer vision algorithm evaluation data set in the automatic driving scene), in combination with the situation in the actual working scene of the tractor:
TABLE 1 description of the tag formats
The following description is made about the category, the truncation degree, and the occlusion rate in the tag format in table 1, and since obstacles appearing in a common scene of agricultural machinery work mainly include tractors and pedestrians, the category is mainly set to two categories of tractors and pedestrians, and truncation and occlusion occurs in a few situations in a field or in a traveling process of agricultural machinery and is therefore set to 0, but occurs in a hangar.
In the following description of the viewing angles in the label format in table 1, the viewing angle of the object is determined by rotating the object around the Y axis of the camera by the Z axis, where the connecting line from the camera to the center of gravity of the object is a radius, and the angle between the object direction and the X axis of the camera is shown in fig. 5, and the conversion relationship between the angles in fig. 5 is shown as the following formula:
thus, α = β - θ. According to the labeling results of the point cloud training data and the image training data, the central position of the three-dimensional frame of the object is assumed to be X = (X, y, z) in a laser radar coordinate system T And when the coordinate of the camera coordinate system is Y, R is a rotation matrix from the camera to the laser radar, and t is a translation matrix, the conversion relation between the two is as follows:
y = RX + t = TX'; formula (11)
In the formula:
the three-dimensional space direction of the object takes the orientation of the collecting tractor as an original point, namely the positive direction of the X axis of the laser radar is 0 degree, the positive half axis along the Y axis is positive, the negative half axis is negative, and the numerical range is-pi.
In addition, the confidence in the label format in table 1 above is the result obtained by the trained model test, and does not appear in the training set, and the confidence is used to draw a P/R curve, where a higher curve indicates a better result.
Through the operation, a labeled target point cloud data set constructed based on the point cloud training data and the image training data is obtained, so that a preset three-dimensional target detection model is input for training.
And 160, inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
Under the condition of obtaining the target point cloud data set, the electronic equipment inputs the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
The preset three-dimensional target detection model can be selected from various existing three-dimensional target detection models. Such as the PointRCNN model, the PVRCNN + + network model, etc. The method and the device can adopt the PVRCNN + + network model as a preset three-dimensional target detection model, and train the target point cloud data set to obtain the target tractor detection model.
The method comprises the steps of collecting multi-source training data consisting of point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene, carrying out time synchronization and space synchronization on the multi-source training data, and constructing a labeled target point cloud data set based on the image training data and the point cloud training data. And inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model. And finally, detecting the tractor based on the target tractor detection model. The multi-source sensor data of various tractors in the agricultural scene are collected, and the target point cloud data set with the labels and with a large amount of point cloud data is constructed based on the two-dimensional image training data and the three-dimensional point cloud training data of various tractors in the agricultural scene, so that the target point cloud data set with the labels has more kinds of tractor characteristic information, a target tractor detection model obtained by training the target point cloud data set with the labels is suitable for detecting tractors in wider agricultural scenes, and the accuracy of detecting various tractors is improved. The defect that the accuracy rate of detecting a plurality of tractors is reduced due to the fact that only specific tractors can be detected currently is overcome.
In other aspects of the present invention, the multi-source data further includes vehicle pose data, and after the step of time-synchronizing and space-synchronizing the point cloud training data and the image training data, the method further includes:
in other aspects of the present invention, referring to fig. 6, before the step 140 of constructing the labeled target point cloud data set based on the point cloud training data and the image training data after the time synchronization and the spatial synchronization, the method further includes:
and step 130, carrying out distortion correction on the point cloud training data based on the vehicle pose data.
The mechanical laser radar for collecting the point cloud data has the working mode of rotatably scanning the surrounding environment at 360 degrees, the frequency of the collected data is 10Hz, namely the time required for collecting one frame is 100ms, and the coordinates of the points obtained by the laser radar are based on the optical center of the laser radar as the original coordinate point. In a scanning period of a frame, laser points are acquired at different moments, in one period, the pose of a laser radar is changed, a coordinate system is changed, the original points of the laser points are inconsistent, however, the laser radar cannot detect the change of the pose of the laser radar, namely, the coordinate systems at different moments cannot be automatically unified, and finally when the laser radar is spliced into a frame, the shape of an actual object and an object spliced by point clouds have errors, so that distortion is generated. The point cloud distortion can refer to fig. 7, in fig. 7a, when the lidar has no attitude change, the starting point and the end point of one frame of point cloud coincide, the scanned point forms a closed circle, in fig. 7b, when the lidar origin moves from O to O', the scanned frame of data is distorted.
Because the laser radar cannot detect the self pose change, the laser radar can be fused with other sensor information to calculate and acquire the pose information, for example, a combined navigation device is used for solving the problem of distortion generated in the movement process of the laser radar.
In some embodiments of the invention, the distortion correcting the point cloud training data based on the vehicle pose data at step 130 comprises:
131, acquiring corresponding and matched pose information of each laser point in each frame of point cloud acquired by a laser radar based on the vehicle pose data;
and 132, converting the coordinates of each laser point in each frame of point cloud into the coordinates of the first laser point in each frame of point cloud under the coordinate system based on the pose information.
Specifically, the acquisition frequency of the laser radar is 10Hz, and in order to ensure that each laser point acquired by the laser radar can be approximately matched with the corresponding pose information, the acquisition frequency of the combined navigation device is 100Hz. Referring to fig. 8, based on the vehicle pose data, pose information corresponding to each laser point in each frame of point cloud acquired by the laser radar is acquired, each laser point in one frame of point cloud is matched with pose information corresponding to the laser point, then coordinate transformation is performed, and the coordinate of each laser point is converted into the coordinate system of the first laser point, so that distortion of the point cloud is reduced, accuracy of point cloud training data of a tractor is improved, and detection accuracy of a target tractor detection model obtained through point cloud training data training is improved.
It should be noted that, in fig. 8, the difference time being smaller than the difference time in t refers to a time difference between the time when the laser radar collects the laser point and the time when the combined navigation device correspondingly collects the pose information of the tractor. Namely, the combined navigation device correspondingly collects the pose information of the tractor while collecting the laser spot by the laser radar. Because complete simultaneous or synchronous acquisition is practically impossible, a time difference exists between the time when the laser radar acquires the laser point and the time when the combined navigation device correspondingly acquires the tractor pose information. t represents a set threshold for the phase difference time, which may be 5ms, for example.
In other aspects of the present invention, referring to fig. 9, before the step 160 of inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model, the method further includes:
and the electronic equipment extracts a target area with a preset three-dimensional size from each frame of point cloud data of the target point cloud data set.
As the distance increases, the point cloud structure of the laser radar gradually loses the spatial information, the accuracy of target detection at a longer distance is sharply reduced, and the performance of neural network training is affected. Therefore, a target area with a preset three-dimensional size needs to be extracted from each frame of point cloud data of the target point cloud data set, and then the target area with the extracted target area needs to be input into a preset three-dimensional target detection model for training.
In a typical embodiment, the tractor point cloud of the VLP-16 laser radar at 28 meters is found to be sparse in the labeling process, which is not beneficial to the subsequent target detection, and the braking distance of the tractor is about 7.5m when the average speed of the tractor is 25km/h, so that the length (a) of the cuboid is determined to be 25m by taking the value which is 2 times as reference and considering the condition that the tractors move in opposite directions. In the transverse direction, the width (b) of the cuboid is determined to be 3 times of the width, namely 15m, by referring to the maximum width (generally less than 5 m) of a common Beijing machine tool. In the vertical direction, the height (c) of the cuboid is larger than the maximum height of a common tractor, and the value is 5m. Finally, the target area of the preset three-dimensional size is determined to be 25m × 15m × 5m, as shown in fig. 10 (a) -10 (c). FIG. 10 (a) shows a side view of a target area; FIG. 10 (b) shows a top view of the target area; fig. 10 (c) is a perspective view of the target region.
Therefore, before the target point cloud data set is input into a preset three-dimensional target detection model, the point cloud data in the target point cloud data set is denser by performing preprocessing operation of extracting a target area with a preset three-dimensional size from each frame of point cloud data of the target point cloud data set, the data quality of the target point cloud data set is improved, and the detection accuracy of the trained target tractor detection model is improved.
And 152, filtering non-ground point cloud data in the target area by using a ground point cloud denoising method based on a plane model.
In order to further extract the target point cloud from a frame of point cloud in the target area, the ground point cloud denoising method based on the plane model filters the non-ground point cloud, reduces input data for a subsequent preset three-dimensional target detection model, and improves the real-time property of the point cloud data on the basis of saving the storage space occupied by the target point cloud data set.
The following describes the tractor detection apparatus provided by the present invention, and the tractor detection apparatus described below and the tractor detection method described above may be referred to in correspondence with each other.
Referring to fig. 11, a tractor testing apparatus includes:
a point cloud data acquisition module 201, configured to acquire point cloud data of a tractor in an agricultural scene;
a detection result obtaining module 202, configured to obtain a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result;
wherein the target tractor detection model is obtained by:
the data acquisition module 203 is used for acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene;
a synchronization module 204, configured to perform time synchronization and space synchronization on the point cloud training data, the image training data, and the vehicle pose data;
a data set constructing module 205, configured to construct a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization;
and the training module 206 is configured to input the target point cloud data set into a preset three-dimensional target detection model for training, so as to obtain a target tractor detection model.
The method comprises the steps of collecting multi-source training data consisting of point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene, carrying out time synchronization and space synchronization on the multi-source training data, and constructing a labeled target point cloud data set based on the image training data and the point cloud training data. And inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model. And finally, detecting the tractor based on the target tractor detection model. The multi-source sensor data of various tractors in the agricultural scene are collected, and the target point cloud data set with the labels and with a large amount of point cloud data is constructed based on the two-dimensional image training data and the three-dimensional point cloud training data of various tractors in the agricultural scene, so that the target point cloud data set with the labels has more kinds of tractor characteristic information, a target tractor detection model obtained by training the target point cloud data set with the labels is suitable for detecting tractors in wider agricultural scenes, and the accuracy of detecting various tractors is improved. The defect that the accuracy rate of detecting a plurality of tractors is reduced due to the fact that only specific tractors can be detected currently is overcome.
On the basis of the above embodiments, as an alternative embodiment, the data set structure modeling block includes:
the marking module is used for projecting the point cloud training data onto a target image and marking the target image and the point cloud training data in the corresponding visual field range of the target image simultaneously by using a double match semi-automatic marking tool; the target image is image training data that is time-synchronized and space-synchronized with the point cloud training data;
the label setting module is used for carrying out label setting on the labeling result in a preset label form to obtain a target point cloud data set with a label; the preset label form at least comprises a two-dimensional boundary frame, a three-dimensional frame size, a three-dimensional frame position and an observation angle of the tractor.
On the basis of the above embodiments, as an optional embodiment, the tractor detection device further includes:
and the point cloud distortion correction module is used for carrying out distortion correction on the point cloud training data based on the vehicle pose data.
On the basis of the foregoing embodiments, as an optional embodiment, the point cloud distortion correcting module includes:
the position and pose information acquisition module is used for acquiring position and pose information which is correspondingly matched with each laser point in each frame of point cloud acquired by the laser radar based on the vehicle position and pose data;
and the coordinate conversion module is used for converting the coordinate of each laser point in each frame of point cloud into the coordinate of the first laser point in each frame of point cloud under the coordinate system based on the pose information.
On the basis of the above embodiments, as an alternative embodiment, the tractor detection device further includes:
the target area extraction module is used for extracting a target area with a preset three-dimensional size from each frame of point cloud data of the target point cloud data set;
and the point cloud filtering module is used for filtering non-ground point cloud data in the target area by a ground point cloud denoising method based on the plane model.
Fig. 12 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 12: a processor (processor) 1212, a communication Interface (Communications Interface) 1220, a memory (memory) 1230, and a communication bus 1240, wherein the processor 1212, the communication Interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. Processor 1212 may invoke logic instructions in memory 1230 to perform a tractor detection method comprising: acquiring point cloud data of a tractor in an agricultural scene; obtaining a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result; the target tractor detection model is obtained through the following steps: acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene; performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data; constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization; and inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
In addition, the logic instructions in the memory 1230 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
On the other hand, referring to fig. 13, the present invention further provides a tractor detection system 100, which includes a laser radar 10 for collecting point cloud training data, a camera 20 for collecting image training data, a combined navigation device 30 for collecting vehicle pose data, and the electronic device 40.
Among them, 16-line lidar VLP-16 available from Velodyne may be used as the lidar 10 of the present invention. The horizontal field angle of the 16-line laser radar VLP-16 is 360 degrees, the vertical field angle can reach 30 degrees (+/-15 degrees), the laser beam is 16 layers with small shape and simple installation mode, and is suitable for three-dimensional perception in a low-speed environment, as the average speeds of the tractor in a tractor plowing channel and a farmland are respectively 28km/h and 10km \ h, and when the acquisition frequency is 10HZ, 288000 data points can be obtained every second, the acquisition requirement of the tractor point cloud is met. Therefore, the laser radar is selected as a three-dimensional perception sensor of the tractor. The device is installed at a position 1.28m away from the ground, a coordinate system of the device is shown in fig. 10 (a), an X axis is the advancing direction of a vehicle, and the relation between the ground height of a laser radar and the detection height of a blind boundary point is as follows:
wherein h is the height from the ground and the unit is m; α — vertical field angle, in °; l is the ground blind zone distance, and the unit is m; h-blind spot detection height in m. The height of the data acquisition tractor is 2.7m, H is 0.89m, H is 3.32m after calculation, and the point cloud data of the common tractor can be acquired.
The camera 20 of the present invention can be an industrial camera manufactured by Shanghai Saint Emerson Vision technology, inc. and having a model number FLIR BFS-PGE-23S3, the resolution of the camera is 1920X1200 (230 ten thousand pixels), the maximum compression frame rate can reach 80 frames per second, the frame rate under full resolution is 53 frames per second, and the camera supports the acquisition mode of single frame and continuous multiple frames. The frequency of image acquisition is 20HZ, the Z-axis direction of the acquired coordinate system is the vehicle advancing direction, the X-axis is right, and the Y-axis is downward.
The integrated navigation device 30 of the present invention can be a Huazhong CGI-610 centimeter-level integrated navigation device, and the positioning and orienting antenna is installed on the roof of the tractor. This integrated navigation device 30 combines together satellite positioning sensor and inertial sensor, adopts the directional GNSS technique of high accuracy location, support 555 passageways, and adopt 2.5 degrees zero-offset high accuracy gyroscope and accelerometer, support external odometer information to carry out the auxiliary computation, with the help of multisensor fusion technique, reliability and dynamic have been improved, can provide high accuracy carrier position, gesture, speed and sensor information in real time, the good application demand who satisfies long-time, high accuracy, high reliability navigation under the complex environment such as agricultural machinery operation environment. The Y-axis direction is the advancing direction of the vehicle, the X-axis is towards the right, and the Z-axis is upwards.
The electronic equipment 40 of the invention can select a Nuvo-810GC industrial personal computer, and is provided with a display screen for visualization. The industrial personal computer consists of
E or 9 th/8 th generation Core. The connection modes between the laser radar 10, the camera 20, the integrated navigation device 30 and the industrial personal computer (i.e., the electronic device 40) are an ethernet UDP protocol, an ethernet UDP protocol and a serial port, respectively.
The method comprises the steps of collecting point cloud training data of various tractors in an agricultural scene through a laser radar, collecting image training data of the various tractors in the agricultural scene through a camera, collecting vehicle pose data of the various tractors in the agricultural scene through a combined navigation device, carrying out time synchronization and space synchronization on multi-source training data through an industrial personal computer, constructing a target point cloud data set with labels based on the image training data and the point cloud training data, inputting the target point cloud data set into a preset three-dimensional target detection model for training, and obtaining a target tractor detection model. And finally, detecting the tractor based on the target tractor detection model. The method is characterized in that multi-source sensor data of various tractors in an agricultural scene are collected, and a target point cloud data set with labels and a large amount of point cloud data is constructed based on two-dimensional image training data and three-dimensional point cloud training data of various tractors in the agricultural scene, so that the target point cloud data set with labels has more kinds of tractor characteristic information, a target tractor detection model obtained by training the target point cloud data set with labels is suitable for detecting tractors in wider agricultural scenes, and the accuracy of detecting various tractors is improved. The defect that the accuracy rate of detecting tractors with various types is reduced due to the fact that only specific tractors can be detected currently is overcome.
In another aspect, the present invention also provides a computer program product comprising a computer program storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of executing the tractor detection method provided by the above methods, the method comprising: acquiring point cloud data of a tractor in an agricultural scene; obtaining a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result; the target tractor detection model is obtained through the following steps: acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene; performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data; constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization; and inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a tractor detection method provided by performing the above methods, the method comprising: acquiring point cloud data of a tractor in an agricultural scene; obtaining a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result; wherein the target tractor detection model is obtained by the following steps: acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene; performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data; constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization; and inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the embodiment. One of ordinary skill in the art can understand and implement the method without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may be modified or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions.
Claims (10)
1. A tractor detection method is characterized by comprising the following steps:
acquiring point cloud data of a tractor in an agricultural scene;
obtaining a tractor detection result: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result;
wherein the target tractor detection model is obtained by the following steps:
acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene;
performing time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data;
constructing a labeled target point cloud data set based on the point cloud training data and the image training data after time synchronization and space synchronization;
and inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
2. The tractor detection method of claim 1, wherein the constructing a tagged target point cloud dataset based on the time-synchronized and spatially-synchronized point cloud training data and the image training data comprises:
projecting point cloud training data onto a target image, and simultaneously labeling the target image and the point cloud training data in a visual field range corresponding to the target image by using a double match semi-automatic labeling tool; the target image is image training data that is time-synchronized and space-synchronized with the point cloud training data;
setting labels for the labeling results in a preset label form to obtain a labeled target point cloud data set; the preset label form at least comprises a two-dimensional boundary frame, a three-dimensional frame size, a three-dimensional frame position and an observation angle of the tractor.
3. The tractor detection method of claim 1, wherein prior to the step of constructing the tagged target point cloud dataset based on the time-synchronized and spatially-synchronized point cloud training data and the image training data, further comprising:
and carrying out distortion correction on the point cloud training data based on the vehicle pose data.
4. The tractor detection method of claim 3, wherein the distortion correcting the point cloud training data based on the vehicle pose data comprises:
acquiring pose information corresponding and matched with each laser point in each frame of point cloud acquired by a laser radar based on the vehicle pose data;
and converting the coordinate of each laser point in each frame of point cloud into the coordinate of the first laser point in each frame of point cloud under the coordinate system based on the pose information.
5. The tractor detection method according to claim 1, wherein before the step of inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model, the method further comprises:
extracting a target area with a preset three-dimensional size from each frame of point cloud data of the target point cloud data set;
and filtering non-ground point cloud data in the target area by using a ground point cloud denoising method based on a plane model.
6. A tractor detection device, its characterized in that includes:
the point cloud data acquisition module is used for acquiring point cloud data of the tractor in an agricultural scene;
the detection result acquisition module is used for acquiring the detection result of the tractor: inputting the point cloud data into a target tractor detection model to obtain a tractor detection result;
wherein the target tractor detection model is obtained by:
the data acquisition module is used for acquiring point cloud training data, image training data and vehicle pose data of various tractors in an agricultural scene;
the synchronization module is used for carrying out time synchronization and space synchronization on the point cloud training data, the image training data and the vehicle pose data;
the data set construction module is used for constructing a target point cloud data set with labels based on the point cloud training data and the image training data after time synchronization and space synchronization;
and the training module is used for inputting the target point cloud data set into a preset three-dimensional target detection model for training to obtain a target tractor detection model.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the tractor detection method according to any of claims 1 to 5 are implemented when the program is executed by the processor.
8. A tractor detection system comprising a lidar configured to collect point cloud training data, a camera configured to collect image training data, an integrated navigation device configured to collect vehicle pose data, and the electronic device of claim 7.
9. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the tractor detection method according to any one of claims 1 to 5.
10. A computer program product comprising a computer program, wherein the computer program when executed by a processor implements the steps of a tractor detection method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210134625.1A CN115880500A (en) | 2022-02-14 | 2022-02-14 | Tractor detection method, apparatus, system, device, medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210134625.1A CN115880500A (en) | 2022-02-14 | 2022-02-14 | Tractor detection method, apparatus, system, device, medium, and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115880500A true CN115880500A (en) | 2023-03-31 |
Family
ID=85755808
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210134625.1A Pending CN115880500A (en) | 2022-02-14 | 2022-02-14 | Tractor detection method, apparatus, system, device, medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115880500A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315035A (en) * | 2023-11-30 | 2023-12-29 | 武汉未来幻影科技有限公司 | Vehicle orientation processing method and device and processing equipment |
-
2022
- 2022-02-14 CN CN202210134625.1A patent/CN115880500A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315035A (en) * | 2023-11-30 | 2023-12-29 | 武汉未来幻影科技有限公司 | Vehicle orientation processing method and device and processing equipment |
| CN117315035B (en) * | 2023-11-30 | 2024-03-22 | 武汉未来幻影科技有限公司 | Vehicle orientation processing method and device and processing equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106681353B (en) | The unmanned plane barrier-avoiding method and system merged based on binocular vision with light stream | |
| CN112894832B (en) | Three-dimensional modeling method, three-dimensional modeling device, electronic equipment and storage medium | |
| JP7073315B2 (en) | Vehicles, vehicle positioning systems, and vehicle positioning methods | |
| CN111199578B (en) | Unmanned aerial vehicle three-dimensional environment modeling method based on vision-assisted laser radar | |
| JP6821712B2 (en) | Calibration of integrated sensor in natural scene | |
| JP6484729B2 (en) | Unmanned aircraft depth image acquisition method, acquisition device, and unmanned aircraft | |
| CN108020825B (en) | Fusion calibration system and method for laser radar, laser camera and video camera | |
| CN112836737A (en) | Roadside combined sensing equipment online calibration method based on vehicle-road data fusion | |
| CN109465832A (en) | High-precision vision and the tight fusion and positioning method of IMU and system | |
| CN106529495A (en) | Obstacle detection method of aircraft and device | |
| CN111275750A (en) | Indoor space panoramic image generation method based on multi-sensor fusion | |
| CN112461210B (en) | Air-ground cooperative building surveying and mapping robot system and surveying and mapping method thereof | |
| CN109547769B (en) | Highway traffic dynamic three-dimensional digital scene acquisition and construction system and working method thereof | |
| CN111060924A (en) | SLAM and target tracking method | |
| CN112162297A (en) | Method for eliminating dynamic obstacle artifacts in laser point cloud map | |
| CN116625354B (en) | High-precision topographic map generation method and system based on multi-source mapping data | |
| CN115451948A (en) | Agricultural unmanned vehicle positioning odometer method and system based on multi-sensor fusion | |
| CN112365549A (en) | Attitude correction method and device for vehicle-mounted camera, storage medium and electronic device | |
| CN110986888A (en) | Aerial photography integrated method | |
| CN115880500A (en) | Tractor detection method, apparatus, system, device, medium, and program product | |
| CN115359130A (en) | Radar and camera combined calibration method and device, electronic equipment and storage medium | |
| CN116978009A (en) | Dynamic object filtering method based on 4D millimeter wave radar | |
| CN113327296B (en) | Laser radar and camera online combined calibration method based on depth weighting | |
| CN116684740A (en) | Perception training data generation method, device, computer equipment and storage medium | |
| CN113895482B (en) | Train speed measuring method and device based on trackside equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |