CN113906900B - Sugarcane harvester and method for adjusting position and posture of cutter head of sugarcane harvester based on multi-sensor fusion - Google Patents
Sugarcane harvester and method for adjusting position and posture of cutter head of sugarcane harvester based on multi-sensor fusion Download PDFInfo
- Publication number
- CN113906900B CN113906900B CN202111128153.0A CN202111128153A CN113906900B CN 113906900 B CN113906900 B CN 113906900B CN 202111128153 A CN202111128153 A CN 202111128153A CN 113906900 B CN113906900 B CN 113906900B
- Authority
- CN
- China
- Prior art keywords
- information
- sugarcane
- vehicle body
- pose
- sugarcane harvester
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D45/00—Harvesting of standing crops
- A01D45/10—Harvesting of standing crops of sugar cane
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
- G01C21/1652—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments with ranging devices, e.g. LIDAR or RADAR
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
- G06T17/205—Re-meshing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Electromagnetism (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Computer Networks & Wireless Communication (AREA)
- Automation & Control Theory (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Environmental Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Harvesting Machines For Specific Crops (AREA)
- Harvester Elements (AREA)
Abstract
The invention discloses a sugarcane harvester, wherein a body of the sugarcane harvester is provided with a plurality of sensors such as a multi-line laser radar, an inertial sensor, a laser range finder and a pressure sensor. The invention also discloses a sugarcane harvester cutter pose adjusting method based on multi-sensor fusion, which comprises the steps of extracting the vehicle body pose and the surrounding environment information of the sugarcane harvester through various sensors, obtaining the position information of a region to be cut and the pose information of the sugarcane harvester after processing, determining the most suitable cutting pose of the cutter according to the cutting requirement of the sugarcane, transmitting a cutter cutting pose instruction to a hydraulic system by a central controller, and driving a hydraulic rod by the hydraulic system to adjust the cutter to achieve the most suitable cutting pose, so that the cutting quality is improved, the sugarcane top-breaking rate is reduced, and the automation and intelligence level of mechanical equipment of the sugarcane harvester is greatly improved.
Description
Technical Field
The invention relates to the technical field of agricultural mechanical equipment, in particular to a sugarcane harvester and a method for adjusting the position and posture of a cutter head of the sugarcane harvester based on multi-sensor fusion.
Background
In recent years, with the continuous development of Chinese economy, the development of the sucrose industry in south China is rapid, so that the planting area of the sugar cane is continuously enlarged, and meanwhile, higher requirements are put forward on the harvest quality of the sugar cane. Among all the steps of harvesting sugarcane, cutting is one of the most difficult steps to master. The sugarcane cutter is a key part for ensuring the sugarcane harvester to improve the cutting quality and reduce the breakage rate of the sugarcane. The size of the head breaking rate of the sugarcane directly influences the germination rate of the perennial root. The difficulty in improving cutting quality and reducing the breakage rate of the sugarcane lies in the control of the posture of the cutter. To the problem, a plurality of domestic research institutions currently carry out research to a certain extent on sugarcane harvesters, such as: the rotation adjustment self-balancing chassis of the sugarcane harvester and the sugarcane harvester are developed by the medium-weight-connected machine limited company, so that the whole existing sugarcane harvester can adapt to the hilly terrain, but cannot adapt to the pavement with a large gradient and a fast change well, and the automation degree is low; the Henan university of science and technology develops a simulation device for adjusting the cutting height of the cutter head through the height of the cutter head, but the operation is simple, and the height of the cutter head can be adjusted up and down only manually; chinese agricultural university has developed a sugarcane harvester furrow bilateral ground profile modeling control system, can realize real-time profile modeling, has further improved the profile modeling precision, but also can not realize the multi-angle regulation to the cutterbar.
The environment of the sugarcane field in China is complex mainly in the hilly areas in the south, and the adjustment of the cutting angle of the cutter of the sugarcane harvester on the terrain change is also very complex. At present, the working mode of an actuating mechanism of harvesting equipment of a sugarcane harvester is generally that the harvesting equipment is adjusted manually according to the condition that human eyes observe a road surface and sugarcane, and the influence of terrain change on cutting is reduced by adopting a ground profiling chassis. The disadvantages of the prior art are mainly three: firstly, a plurality of visual field blind areas exist in the sugarcane harvesting process, time and labor are consumed through a manual adjusting mode, the requirement on the proficiency of operators is high, the efficiency is not high, and the precision is difficult to guarantee; secondly, the existing products and technologies can only adjust the height of the harvesting equipment, and once the harvester deflects in a non-horizontal direction, the cutter disc can not reach an ideal cutting pose only by adjusting the height direction; thirdly, the ground profiling device can only reduce the influence of the change of the terrain on cutting, is relatively slow to adjust and cannot realize accurate control on the pose of the cutter head. The prior art can not well improve the cutting quality of the sugarcane and reduce the breakage rate of the sugarcane.
Disclosure of Invention
The invention aims to provide a sugarcane harvester and a method for adjusting the pose of a cutter head of the sugarcane harvester based on multi-sensor fusion, which are used for transforming the traditional sugarcane harvester and can intelligently control and adjust the pose of the cutter head in real time.
In order to achieve the purpose, the invention discloses a sugarcane harvester which comprises a cutter head, a hydraulic rod, a hydraulic system and a control system, wherein the cutter head is positioned at the bottom of the sugarcane harvester.
The hydraulic rod is connected with the cutter head, the hydraulic system is connected with the hydraulic rod, and the hydraulic system controls the extension and retraction of the hydraulic rod.
The hydraulic rods comprise a first hydraulic rod vertical to the horizontal plane, and further comprise a second hydraulic rod, a third hydraulic rod, a fourth hydraulic rod and a fifth hydraulic rod on the horizontal plane; the first hydraulic rod is used for controlling the movement of the cutter head in the vertical direction, and the second hydraulic rod, the third hydraulic rod, the fourth hydraulic rod and the fifth hydraulic rod are used for controlling the movement of the cutter head in the horizontal direction.
The control system comprises a multi-line laser radar arranged at the top of the sugarcane harvester, an inertial sensor arranged in the sugarcane harvester, a laser range finder, a pressure sensor arranged above wheels and a central controller; the laser range finder is arranged at the bottom of the sugarcane harvester and is positioned in front of the cutter head; the central controller is respectively connected with the multi-line laser radar, the inertial sensor, the laser range finder and the pressure sensor, and is responsible for collecting and processing data transmitted by the multi-line laser radar, the inertial sensor, the laser range finder and the pressure sensor and sending a control command to the hydraulic system to control the cutting attitude of the cutter head.
Furthermore, the multi-line laser radar is rigidly connected with the top of the sugarcane harvester, and the inertial sensor is rigidly connected with the sugarcane harvester.
The method for adjusting the position and the attitude of the cutter head of the sugarcane harvester based on the multi-sensor fusion comprises the following steps:
step one, a plurality of sensors carry out data acquisition and pretreatment, and the method comprises the following steps: acquiring point cloud information of the surrounding environment of the sugarcane harvester by using a multi-line laser radar to obtain x, y and z coordinates of each point cloud; the method comprises the following steps that an inertial sensor collects vehicle body angle information and acceleration information of the sugarcane harvester; the laser range finder continuously measures the distance to the ground according to the change of the terrain, and generates a oscillogram of the distance from the laser range finder to the ground in real time; the pressure sensor collects the pressure value of the vehicle body to the wheels;
secondly, carrying out sugarcane target identification by using a convolutional neural network, and carrying out semantic segmentation on the model structure of the ground and the sugarcane canopy to obtain position information to be cut;
thirdly, performing interframe matching according to the edge information and the plane information of the point cloud to calculate the position information and the posture information of the sugarcane harvester;
step four, performing pre-integration processing on the inertial sensor to obtain vehicle body position information and attitude information of the sugarcane harvester, resolving the attitude of the vehicle body by combining road surface gradient information output by the pressure sensor after processing, and estimating and predicting the subsequent attitude;
step five, performing fusion complementation on the obtained data by adopting a mode based on a graph model, and outputting the position information and the posture information of the sugarcane harvester at the current moment in real time;
and step six, after the steps are carried out, accurate vehicle body pose information and position information of a part to be cut can be obtained, the most appropriate cutting posture of the cutter head can be determined by combining the cutting requirements of the sugarcane, the central controller transmits a cutter head cutting posture instruction to the hydraulic system, and the hydraulic system drives the hydraulic rod to adjust the cutter head to achieve the most appropriate cutting posture, so that the subsequent cutting work is completed.
Further, the second step of using a convolutional neural network to perform sugarcane target identification, performing semantic segmentation on the model structure of the ground and the sugarcane canopy, and acquiring the position information to be cut comprises the following steps:
firstly, after a convolutional neural network is used for carrying out sugarcane target identification, semantic segmentation is carried out on surrounding point cloud by adopting an improved Randlanet, and a ground area and a sugarcane field area to be harvested are segmented; when semantic segmentation is performed on surrounding point clouds by using an improved Randlanet, a data set is established for labeling training, weight parameters are adjusted, and then semantic segmentation is performed, so that the segmentation accuracy and robustness can be effectively improved;
after the ground area and the sugarcane field area are segmented, three-dimensional grid division is carried out on the segmented areas, then a voxel filter is adopted to filter surrounding point clouds, and then line characteristic points and surface characteristic points are respectively segmented on the ground area and the sugarcane field area; the method for dividing and taking the line characteristic points and the surface characteristic points comprises the following steps: the coordinates of ten laser points on the same laser line of the current point cloud are respectively subtracted from the coordinates of the point to be solved, and then the module is respectively taken and the summation is carried out; wherein, the point with the maximum numerical value is taken as the point with the maximum curvature on the laser line and is taken as the characteristic point of the line; the two points with the minimum numerical value are taken as the two points with the minimum curvature on the laser line and are taken as surface characteristic points; setting threshold values for other points according to actual conditions to divide the points into secondary line characteristic points and secondary surface characteristic points so as to divide line characteristic points and surface characteristic points on 16 laser lines;
selecting a certain amount of line characteristic points and surface characteristic points in the fixed area to start dividing the edge area and the plane area, further refining and dividing the point cloud according to the detected edge area and the detected plane area, fitting a three-dimensional contour model of a canopy area and the ground of the sugarcane field, and calculating the height difference between the canopy with the same section and the ground so as to deduce the height information of the sugarcane canopy and the position to be cut.
Furthermore, because the multi-line laser radar has a certain shielding object during scanning, and a scanned ground model is not necessarily complete, when the height information of the canopy and the position to be cut are deduced in the step two, the optimization and the perfection are carried out by adopting a data fusion mode of the laser range finder and the multi-line laser radar aiming at the area with the incomplete model, the vertical distance between the laser range finder and the multi-line laser radar is calculated according to the space geometric relation of the laser range finder and the multi-line laser radar, the vertical distance between the multi-line laser radar and the ground can be obtained according to the vertical distance between the laser range finder and the ground, and finally the height information of the sugarcane can be calculated according to the vertical distance between the multi-line laser radar and the canopy of the sugarcane and the position to be cut can be deduced.
Further, the method for calculating the position information and the posture information of the sugarcane harvester by performing frame-to-frame matching according to the edge information and the plane information of the point cloud comprises the following steps:
searching and matching edge feature points and plane feature points between a current frame and a previous frame of point cloud by using kdtree, performing pose transformation on successfully matched point cloud to enable feature coincidence of two frames of point cloud, so as to calculate the pose condition of the multi-line laser radar at the current moment, then iterating the calculated pose at the current moment of the multi-line laser radar by using an LM algorithm, continuously reducing error and correcting pose parameters of the multi-line laser radar, and using the pose parameters of the multi-line laser radar as vehicle body pose parameters of the sugarcane harvester;
the position information of the vehicle body of the sugarcane harvester can be estimated by continuously updating the position and attitude information of the multi-line laser radar, and the position and attitude parameters of the vehicle body and the position information of the vehicle body are transmitted to the central controller in real time to monitor the position of the sugarcane harvester in real time.
Further, the method for calculating the attitude of the vehicle body by performing pre-integration processing on the inertial sensor to obtain the vehicle body position information and the attitude information of the sugarcane harvester and combining the road surface gradient information output by the pressure sensor after processing comprises the following steps:
the pressure obtained by the pressure sensor is compared with the pressure on a horizontal road, the gradient information of the road surface is obtained through calculation, then the inertial sensor is subjected to pre-integration processing to improve the calculation efficiency, the angle parameters of the vehicle body and the horizontal plane of the sugarcane harvester can be obtained according to the angle parameters output by the inertial sensor, and the inclination angle information of the vehicle body and the horizontal plane can be preliminarily calculated;
and then, according to the road surface gradient information output after the processing of the pressure sensor, combining the inclination angle information of the vehicle body and the horizontal plane, and accurately calculating the angle relationship between the vehicle body and the ground.
Further, the method for estimating and predicting the subsequent attitude in the fourth step comprises the following steps:
and estimating and predicting the pose of the vehicle body at the next moment according to the acceleration information provided by the ground contour model and the inertial sensor segmented by the point cloud, and providing initial value information with enough robustness for the pose information of the vehicle body at the next moment so as to quickly respond to the next control of the cutter head.
Further, the fifth step of adopting a mode based on a graph model to perform fusion complementation on the obtained data, and outputting the vehicle body position information and the posture information of the sugarcane harvester at the current moment in real time comprises the following steps:
firstly, taking the parameters measured in the fourth step as the initial pose of the vehicle body, wherein the parameters are the angle relation between the vehicle body and the ground and the acceleration of the vehicle body, adding the pose information of the vehicle body obtained by the multi-line laser radar and the height information measured by the laser range finder every 5s of continuous output of the inertial sensor, correcting the pose once, eliminating the accumulated error of the inertial sensor in the long-time use condition, continuously repeating the process and finally outputting the position information and the pose information of the vehicle body of the sugarcane harvester at the current moment in real time.
Compared with the prior art, the invention has the following beneficial effects:
(1) According to the invention, various sensors can be additionally arranged or modified on the basis of the chassis of the sugarcane harvester, so that the technical cost is reduced, and the automatic adjustment of the cutting posture of the cutter head is realized;
(2) The invention uses a multi-line laser radar to extract point cloud information of the surrounding environment, uses an inertial sensor to extract angle information and acceleration information of a sugarcane harvester body, uses a laser range finder to continuously measure the distance to the ground, uses a pressure sensor to extract the pressure value of the body and the ground, then uses a deep neural network to perform target detection on the scanned point cloud to obtain position information and shape information of a target, then uses the multi-line laser radar, the inertial sensor and the like to perform measurement and calculation on pose information of the body and gradient information of the ground, then fuses the information extracted by different sensors based on a graph model, obtains the position information of a region to be cut and the pose information of the sugarcane harvester after processing, determines the most suitable cutting pose of a cutter head by combining cutting requirements of the sugarcane, transmits a cutter head cutting pose instruction to a hydraulic system by a central controller, drives a hydraulic rod to adjust the cutter head so as to achieve the most suitable cutting pose, thereby improving the cutting quality, reducing the sugarcane head breakage rate and greatly improving the automation and intelligent level of mechanical equipment of the sugarcane harvester.
Drawings
FIG. 1 is a schematic structural view of a sugar cane harvester of the present invention;
FIG. 2 is a schematic view of the hydraulic stem structure of the cane harvester of the present invention;
FIG. 3 is a flow chart of a method for adjusting the pose of a cutter head of a sugarcane harvester based on multi-sensor fusion, disclosed by the invention;
FIG. 4 is a schematic diagram of multi-sensor data fusion based on graph models.
In the figure, 1-a multi-line laser radar, 2-a central controller, 3-a first pressure sensor, 4-a second pressure sensor, 5-an inertial sensor, 6-a cutter head, 7-a laser range finder, 8-a first hydraulic rod, 9-a second hydraulic rod, 10-a third hydraulic rod, 11-a fourth hydraulic rod and 12-a fifth hydraulic rod.
Detailed Description
The following detailed description of the present invention is provided in conjunction with the accompanying drawings, but it should be understood that the scope of the present invention is not limited to the specific embodiments.
Throughout the specification and claims, unless explicitly stated otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element or component but not the exclusion of any other element or component.
The embodiment provides a sugarcane harvester and a method for adjusting the position and posture of a cutter head of the sugarcane harvester based on multi-sensor fusion.
Fig. 1 shows a schematic structural view of a sugar cane harvester according to a preferred embodiment of the present invention, comprising a cutter head 6, hydraulic rods, a hydraulic system and a control system, said cutter head 6 being located at the bottom of the sugar cane harvester;
the hydraulic rod is connected with the cutter head 6, the hydraulic system is connected with the hydraulic rod, and the hydraulic system controls the extension and retraction of the hydraulic rod;
fig. 2 shows a schematic structural diagram of the hydraulic rods, which includes a first hydraulic rod 8 perpendicular to the horizontal plane, and further includes a second hydraulic rod 9, a third hydraulic rod 10, a fourth hydraulic rod 11 and a fifth hydraulic rod 12 on the horizontal plane; the first hydraulic rod 8 is used for controlling the movement of the cutter head 6 in the vertical direction, and the second hydraulic rod 9, the third hydraulic rod 10, the fourth hydraulic rod 11 and the fifth hydraulic rod 12 are used for controlling the movement of the cutter head 6 in the horizontal direction;
the control system comprises a multi-line laser radar 1 arranged at the top of the sugarcane harvester, an inertial sensor 5 arranged in the sugarcane harvester, a laser range finder 7, a pressure sensor arranged above wheels and a central controller 2; the laser range finder 7 is arranged at the bottom of the sugarcane harvester and is positioned in front of the cutter head 6; the central controller 2 is respectively connected with the multi-line laser radar 1, the inertial sensor 5, the laser range finder 7 and the pressure sensor, and the central controller 2 is responsible for collecting and processing data transmitted by the multi-line laser radar 1, the inertial sensor 5, the laser range finder 7 and the pressure sensor and then transmitting a control instruction to the hydraulic system to control the cutting attitude of the cutter head;
wherein the number of pressure sensors is 4, and the pressure sensors are respectively arranged above 4 wheels, and a first pressure sensor 3 and a second pressure sensor 4 which are positioned on the same side are shown in fig. 1; the multi-line laser radar 1 is rigidly connected with the top of the sugarcane harvester, and the inertial sensor 5 is rigidly connected with the sugarcane harvester.
Fig. 3 shows a flow chart of the method for adjusting the pose of the cutter head of the sugarcane harvester based on multi-sensor fusion, and referring to fig. 3, the method for adjusting the pose of the cutter head comprises the following steps:
step one, a plurality of sensors carry out data acquisition and pretreatment, and the method comprises the following steps: the multi-line laser radar 1 collects point cloud information of the surrounding environment of the sugarcane harvester to obtain x, y and z coordinates of each point cloud; the inertial sensor 5 acquires the vehicle body angle information and the acceleration information of the sugarcane harvester; the laser range finder 7 continuously measures the distance to the ground according to the change of the terrain and generates a oscillogram of the distance from the laser range finder to the ground in real time; 4 pressure sensors measure the pressure values of the vehicle body to four wheels;
the central controller 2 receives data of each sensor, is responsible for carrying out fusion processing on the data, and sends a control instruction to the hydraulic system to carry out pose control on the cutterhead 6.
Secondly, carrying out sugarcane target identification by using a convolutional neural network, carrying out semantic segmentation on the model structure of the ground and the sugarcane canopy, and acquiring position information to be cut:
firstly, after a convolutional neural network is used for carrying out sugarcane target identification, semantic segmentation is carried out on surrounding point cloud by adopting an improved Randlanet, and a ground area and a sugarcane field area to be harvested are segmented; when semantic segmentation is performed on surrounding point clouds by using the improved Randlanet, a data set is established for labeling training, weight parameters are adjusted, and then the semantic segmentation is performed, so that the segmentation accuracy and robustness can be effectively improved;
after the ground area and the sugarcane field area are segmented, three-dimensional grid division is carried out on the segmented areas, then a voxel filter is adopted to filter surrounding point clouds, and then line characteristic points and surface characteristic points are respectively segmented on the ground area and the sugarcane field area; the method for dividing and taking the line characteristic points and the surface characteristic points comprises the following steps: the coordinates of ten laser points on the same laser line of the current point cloud are respectively subtracted from the coordinates of the point to be solved, and then the module is respectively taken and the summation is carried out; wherein, the point with the maximum numerical value is taken as the point with the maximum curvature on the laser line and is taken as a line characteristic point; two points with the minimum numerical value are taken as two points with the minimum curvature on the laser line and are taken as surface characteristic points; setting threshold values for other points according to actual conditions to divide the points into secondary line characteristic points and secondary surface characteristic points so as to divide line characteristic points and surface characteristic points on 16 laser lines;
selecting a certain amount of line characteristic points and surface characteristic points in a fixed area to start dividing an edge area and a plane area, further refining and dividing the point cloud according to the detected edge area and plane area, fitting a three-dimensional contour model of a canopy area and the ground of the sugarcane field, and calculating the height difference between the canopy and the ground of the same section so as to deduce the height information of the sugarcane canopy and the position to be cut;
because the multiline laser radar 1 has a certain shielding object during scanning, a scanned ground model is not complete, and therefore when the height information of a canopy and a position to be cut are deduced, the data fusion mode of the laser range finder 7 and the multiline laser radar 1 is adopted for optimizing and perfecting aiming at the area with the incomplete model, the vertical distance between the laser range finder 7 and the multiline laser radar 1 is calculated according to the space geometric relation of the laser range finder 7 and the multiline laser radar 1, the vertical distance between the multiline laser radar 1 and the ground can be obtained according to the vertical distance between the multiline laser range finder 7 and the ground, and finally the height information of the sugarcane can be calculated according to the vertical distance between the multiline laser radar 1 and the sugarcane canopy, and the position to be cut is deduced.
Step three, performing interframe matching according to the edge information and the plane information of the point cloud to calculate the position information and the posture information of the sugarcane harvester:
searching and matching edge feature points and plane feature points between a current frame and a previous frame of point cloud by adopting kdtree, performing pose transformation on the successfully matched point cloud to ensure that two frames of point cloud are subjected to feature coincidence so as to calculate the pose condition of the multi-line laser radar 1 at the current moment, then iterating the calculated pose of the multi-line laser radar 1 at the current moment by adopting an LM algorithm, continuously reducing error and correcting the pose parameters of the multi-line laser radar 1, and taking the pose parameters of the multi-line laser radar 1 as the vehicle body pose parameters of the sugarcane harvester;
the position information of the vehicle body of the sugarcane harvester can be estimated by continuously updating the position information of the multi-line laser radar 1, the position parameters of the vehicle body and the position information of the vehicle body are transmitted to the central controller 2 in real time, the position of the sugarcane harvester is monitored in real time, and damage to an area which is not harvested can be effectively prevented.
Step four, pre-integrating the inertial sensor 5 to obtain the position information and the attitude information of the vehicle body of the sugarcane harvester, resolving the attitude of the vehicle body by combining the road slope information output by the pressure sensor after processing, and estimating and predicting the subsequent attitude:
the pressure obtained by the 4 pressure sensors is compared with the pressure on a horizontal road surface, the gradient information of the road surface is obtained through calculation, then the inertia sensor 5 is subjected to pre-integration processing to improve the calculation efficiency, the angle parameters of the vehicle body and the horizontal plane of the sugarcane harvester can be obtained according to the angle parameters output by the gyroscope in the inertia sensor 5, and the dip angle information of the vehicle body and the horizontal plane can be preliminarily calculated;
because the vehicle body is not rigidly connected with the wheels, and a buffer device is arranged between the vehicle body and the wheels, the pose of the vehicle body is not very accurate only by the slope information of the ground, and the angle relation between the vehicle body and the ground can be accurately calculated by combining the slope information of the vehicle body and the horizontal plane according to the road slope information output after the processing of the pressure sensor;
and estimating and predicting the pose of the vehicle body at the next moment according to the ground contour model segmented by the point cloud and the acceleration information provided by the inertial sensor 5, and providing initial value information with enough robustness for the pose information of the vehicle body at the next moment so as to make quick response to the next control of the cutter head.
And step five, performing fusion complementation on the obtained data by adopting a mode based on a graph model, and outputting the position information and the posture information of the sugarcane harvester at the current moment in real time:
the inertial sensor has the characteristic of high frequency, so that the inertial sensor is selected as a sensor for continuously outputting the pose of the vehicle body. However, certain accumulated errors can occur when the inertial sensor is used for a long time, and the vehicle body pose information obtained by the low-frequency multi-line laser radar and the height change information measured by the laser range finder can be used as correction factors of the errors of the inertial sensor to eliminate the errors;
as shown in fig. 4, the parameters measured in the fourth step are used as the initial pose of the vehicle body, the parameters are the angular relationship between the vehicle body and the ground and the acceleration of the vehicle body, the inertial sensor 5 adds the pose information of the vehicle body obtained by the multi-line laser radar 1 and the height information measured by the laser range finder 7 every 5 seconds, the pose is corrected for the first time, the accumulated error of the inertial sensor 5 in the long-term use condition is eliminated, and the repeated process is not repeated, so that the position information and the pose information of the vehicle body of the sugarcane harvester at the current moment are finally output in real time;
in fig. 4, the Imu factor refers to a parameter measured by the inertial sensor.
Step six, accurate vehicle body pose information and position information of a part to be cut of the sugarcane can be obtained, the cutting requirement of the sugarcane is combined, the most appropriate cutting posture of the cutter head 6 can be determined, the central controller 2 transmits a cutting posture instruction of the cutter head 6 to a hydraulic system, and the hydraulic system drives a hydraulic rod to adjust the cutter head 6 to achieve the most appropriate cutting posture, so that subsequent cutting work is completed.
The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and its practical application to enable one skilled in the art to make and use various exemplary embodiments of the invention and various alternatives and modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.
Claims (7)
1. A sugarcane harvester cutter head pose adjusting method based on multi-sensor fusion is characterized by comprising the following steps:
the sugarcane harvester comprises a cutter head, a hydraulic rod, a hydraulic system and a control system, wherein the cutter head is positioned at the bottom of the sugarcane harvester;
the hydraulic rod is connected with the cutter head, the hydraulic system is connected with the hydraulic rod, and the hydraulic system controls the extension and retraction of the hydraulic rod;
the hydraulic rods comprise a first hydraulic rod vertical to the horizontal plane, and further comprise a second hydraulic rod, a third hydraulic rod, a fourth hydraulic rod and a fifth hydraulic rod on the horizontal plane; the first hydraulic rod is used for controlling the movement of the cutter head in the vertical direction, and the second hydraulic rod, the third hydraulic rod, the fourth hydraulic rod and the fifth hydraulic rod are used for controlling the movement of the cutter head in the horizontal direction;
the control system comprises a multi-line laser radar arranged at the top of the sugarcane harvester, an inertial sensor arranged in the sugarcane harvester, a laser range finder, a pressure sensor arranged above wheels and a central controller; the laser range finder is arranged at the bottom of the sugarcane harvester and is positioned in front of the cutter head; the central controller is respectively connected with the multi-line laser radar, the inertial sensor, the laser range finder and the pressure sensor, and is responsible for collecting and processing data transmitted by the multi-line laser radar, the inertial sensor, the laser range finder and the pressure sensor and then transmitting a control command to the hydraulic system to control the cutting attitude of the cutter head;
the multi-line laser radar is rigidly connected with the top of the sugarcane harvester, and the inertial sensor is rigidly connected with the sugarcane harvester;
the adjusting method comprises the following steps:
step one, a plurality of sensors carry out data acquisition and pretreatment, and the method comprises the following steps: acquiring point cloud information of the surrounding environment of the sugarcane harvester by using a multi-line laser radar to obtain x, y and z coordinates of each point cloud; the method comprises the following steps that an inertial sensor collects vehicle body angle information and acceleration information of the sugarcane harvester; the laser range finder continuously measures the distance to the ground according to the change of the terrain, and generates a oscillogram of the distance from the laser range finder to the ground in real time; the pressure sensor collects the pressure value of the vehicle body to the wheels;
secondly, carrying out sugarcane target identification by using a convolutional neural network, and carrying out semantic segmentation on the model structure of the ground and the sugarcane canopy to obtain position information to be cut;
thirdly, performing interframe matching according to the edge information and the plane information of the point cloud to calculate the position information and the posture information of the sugarcane harvester;
step four, performing pre-integration processing on the inertial sensor to obtain vehicle body position information and attitude information of the sugarcane harvester, resolving the attitude of the vehicle body by combining road surface gradient information output by the pressure sensor after processing, and estimating and predicting the subsequent attitude;
step five, performing fusion complementation on the obtained data by adopting a mode based on a graph model, and outputting the position information and the posture information of the sugarcane harvester at the current moment in real time;
and step six, after the steps are carried out, accurate vehicle body pose information and position information of a part to be cut can be obtained, the most appropriate cutting posture of the cutter can be determined by combining cutting requirements of the sugarcane, a central controller transmits a cutter cutting posture instruction to a hydraulic system, and the hydraulic system drives a hydraulic rod to adjust the cutter so as to achieve the most appropriate cutting posture, so that subsequent cutting work is completed.
2. The adjusting method according to claim 1, wherein in the second step, the convolutional neural network is used for carrying out target recognition on the sugarcane, semantic segmentation is carried out on the model structure of the ground and the sugarcane canopy, and the method for acquiring the position information to be cut comprises the following steps:
firstly, after a convolutional neural network is used for carrying out sugarcane target identification, semantic segmentation is carried out on surrounding point cloud by adopting an improved Randlanet, and a ground area and a sugarcane field area to be harvested are segmented; when semantic segmentation is performed on surrounding point clouds by using the improved Randlanet, a data set is established for labeling training, weight parameters are adjusted, and then the semantic segmentation is performed, so that the segmentation accuracy and robustness can be effectively improved;
after the ground area and the sugarcane field area are segmented, three-dimensional grid division is carried out on the segmented areas, then a voxel filter is adopted to filter surrounding point clouds, and then line characteristic points and surface characteristic points are respectively segmented on the ground area and the sugarcane field area; the method for dividing and taking the line characteristic points and the surface characteristic points comprises the following steps: the coordinates of ten laser points on the same laser line of the current point cloud are respectively subtracted from the coordinates of the point to be solved, and then the module is respectively taken and the summation is carried out; wherein, the point with the maximum numerical value is taken as the point with the maximum curvature on the laser line and is taken as a line characteristic point; the two points with the minimum numerical value are taken as the two points with the minimum curvature on the laser line and are taken as surface characteristic points; setting threshold values for other points according to actual conditions to divide the points into secondary line characteristic points and secondary surface characteristic points so as to divide line characteristic points and surface characteristic points on 16 laser lines;
selecting a certain amount of line characteristic points and surface characteristic points in the fixed area to start dividing the edge area and the plane area, further refining and dividing the point cloud according to the detected edge area and the detected plane area, fitting a three-dimensional contour model of a canopy area and the ground of the sugarcane field, and calculating the height difference between the canopy with the same section and the ground so as to deduce the height information of the sugarcane canopy and the position to be cut.
3. The adjustment method according to claim 2, characterized in that: and secondly, when the height information of the canopy and the position to be cut are deduced, optimizing and perfecting the area with the incomplete model by adopting a data fusion mode of a laser range finder and a multi-line laser radar, calculating the vertical distance between the laser range finder and the multi-line laser radar according to the space geometric relation of the laser range finder and the multi-line laser radar, obtaining the vertical distance between the multi-line laser radar and the ground according to the vertical distance between the laser range finder and the ground, and finally calculating the height information of the sugarcane according to the vertical distance between the multi-line laser radar and the canopy of the sugarcane and deducing the position to be cut.
4. The adjusting method according to claim 1, wherein the step three of calculating the position information and the posture information of the sugarcane harvester by performing frame-to-frame matching according to the edge information and the plane information of the point cloud comprises the following steps:
searching and matching edge feature points and plane feature points between a current frame and a previous frame of point cloud by using kdtree, performing pose transformation on successfully matched point cloud to enable feature coincidence of two frames of point cloud, so as to calculate the pose condition of the multi-line laser radar at the current moment, then iterating the calculated pose at the current moment of the multi-line laser radar by using an LM algorithm, continuously reducing error and correcting pose parameters of the multi-line laser radar, and using the pose parameters of the multi-line laser radar as vehicle body pose parameters of the sugarcane harvester;
the position information of the vehicle body of the sugarcane harvester can be estimated by continuously updating the position and attitude information of the multi-line laser radar, and the position and attitude parameters of the vehicle body and the position information of the vehicle body are transmitted to the central controller in real time to monitor the position of the sugarcane harvester in real time.
5. The adjusting method according to claim 1, wherein the fourth step is to perform pre-integration processing on the inertial sensor to obtain vehicle body position information and attitude information of the sugarcane harvester, and the method for calculating the attitude of the vehicle body by combining road surface gradient information output by the pressure sensor after processing comprises the following steps:
the pressure obtained by the pressure sensor is compared with the pressure on a horizontal road, the gradient information of the road surface is obtained through calculation, then the inertia sensor is subjected to pre-integration processing to improve the calculation efficiency, the angle parameters of the vehicle body and the horizontal plane of the sugarcane harvester can be obtained according to the angle parameters output by the inertia sensor, and the inclination angle information of the vehicle body and the horizontal plane can be preliminarily calculated;
and then, according to the road surface gradient information output after the processing of the pressure sensor, combining the inclination angle information of the vehicle body and the horizontal plane, and accurately calculating the angle relationship between the vehicle body and the ground.
6. The adjustment method according to claim 5, wherein the estimation and prediction of the subsequent pose in the step four is performed by:
and estimating and predicting the pose of the vehicle body at the next moment according to the acceleration information provided by the ground contour model and the inertial sensor segmented by the point cloud, and providing initial value information with enough robustness for the pose information of the vehicle body at the next moment so as to quickly respond to the next control of the cutter head.
7. The adjusting method according to claim 1, wherein in the fifth step, the obtained data are fused and complemented in a mode based on a graph model, and the method for outputting the vehicle body position information and the posture information of the sugarcane harvester at the current moment in real time comprises the following steps:
firstly, parameters measured in the fourth step are used as the initial pose of the vehicle body, the parameters are the angle relation between the vehicle body and the ground and the acceleration of the vehicle body, the inertial sensor adds the pose information of the vehicle body obtained by the multi-line laser radar and the height information measured by the laser range finder every 5s of continuous output, the pose is corrected for one time, the accumulated error of the inertial sensor under the condition of long-time use is eliminated, and the process of continuous duplication is carried out to finally output the position information and the pose information of the vehicle body of the sugarcane harvester at the current moment in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111128153.0A CN113906900B (en) | 2021-09-26 | 2021-09-26 | Sugarcane harvester and method for adjusting position and posture of cutter head of sugarcane harvester based on multi-sensor fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111128153.0A CN113906900B (en) | 2021-09-26 | 2021-09-26 | Sugarcane harvester and method for adjusting position and posture of cutter head of sugarcane harvester based on multi-sensor fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113906900A CN113906900A (en) | 2022-01-11 |
| CN113906900B true CN113906900B (en) | 2022-11-11 |
Family
ID=79236138
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111128153.0A Active CN113906900B (en) | 2021-09-26 | 2021-09-26 | Sugarcane harvester and method for adjusting position and posture of cutter head of sugarcane harvester based on multi-sensor fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113906900B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114777758A (en) * | 2022-03-21 | 2022-07-22 | 江苏盛海智能科技有限公司 | Initial position determining method and terminal based on laser radar mapping positioning |
| CN114868510B (en) * | 2022-07-08 | 2022-10-11 | 农业农村部南京农业机械化研究所 | Automatic digging depth regulating and controlling method of intelligent fresh potato combine harvester |
| CN115147685B (en) * | 2022-07-20 | 2023-05-05 | 广西大学 | Identification and prediction method for operating state of sugarcane harvester |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106651779A (en) * | 2016-09-09 | 2017-05-10 | 广西大学 | Sugarcane cutting height detection method |
| CN110622692A (en) * | 2019-09-09 | 2019-12-31 | 广西大学 | Intelligent identification method and system for running state of sugarcane combine harvester |
| CN111726378A (en) * | 2019-03-20 | 2020-09-29 | 广西农业机械研究院有限公司 | Remote monitoring system of sugarcane harvester and control method thereof |
| CN113298670A (en) * | 2020-02-06 | 2021-08-24 | 迪尔公司 | Predictive machine map generation and control system |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102057799B (en) * | 2010-10-29 | 2012-07-25 | 广西大学 | Cutter lifting mechanism of sugarcane combined harvester |
| US8452501B1 (en) * | 2011-11-09 | 2013-05-28 | Trimble Navigation Limited | Sugar cane harvester automatic cutter height control |
| US9961832B2 (en) * | 2014-06-13 | 2018-05-08 | Deere & Company | Control of base cutter orientation in sugarcane harvesters |
| CN105612922B (en) * | 2015-12-28 | 2018-03-20 | 广西大学 | Cut quality appraisal procedure and device based on the vibration of cane harvesting machine cutterhead |
| CN106292272B (en) * | 2016-09-17 | 2019-05-31 | 广西大学 | A kind of real-time control method of the cutterhead cutting-in of cane harvesting machine |
| CN107371594A (en) * | 2017-08-24 | 2017-11-24 | 广西民族大学 | A kind of cutterhead dynamic balancing adjusting means of sugar cane cutter |
| CN208079834U (en) * | 2018-02-22 | 2018-11-13 | 长安大学 | A kind of cutter device of the sugar-cane cutting machine of automatic adjusument |
| US20200337232A1 (en) * | 2019-04-24 | 2020-10-29 | Deere & Company | Information inference for agronomic data generation in sugarcane applications |
| CN112056087B (en) * | 2019-06-11 | 2024-01-30 | 中国科学院沈阳自动化研究所 | Induction system of small-sized segmented crawler sugarcane harvester and control method |
| CN110352707B (en) * | 2019-08-23 | 2022-03-22 | 柳州市孚桂智能科技有限公司 | Automatic profiling follow-up system of sugarcane harvester and control method thereof |
| CN111096147B (en) * | 2020-01-15 | 2022-09-06 | 唐山市军星五金工具制造有限责任公司 | Cutter head adjusting device of sugarcane harvester |
-
2021
- 2021-09-26 CN CN202111128153.0A patent/CN113906900B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106651779A (en) * | 2016-09-09 | 2017-05-10 | 广西大学 | Sugarcane cutting height detection method |
| CN111726378A (en) * | 2019-03-20 | 2020-09-29 | 广西农业机械研究院有限公司 | Remote monitoring system of sugarcane harvester and control method thereof |
| CN110622692A (en) * | 2019-09-09 | 2019-12-31 | 广西大学 | Intelligent identification method and system for running state of sugarcane combine harvester |
| CN113298670A (en) * | 2020-02-06 | 2021-08-24 | 迪尔公司 | Predictive machine map generation and control system |
Non-Patent Citations (2)
| Title |
|---|
| 甘蔗收获机切刀负载压力影响因素试验研究;麻芳兰等;《农机化研究》;20181224(第09期);第142-147页 * |
| 甘蔗茎秆切割力的计算;刘庆庭等;《农业机械学报》;20060928(第09期);第89-92页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113906900A (en) | 2022-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113906900B (en) | Sugarcane harvester and method for adjusting position and posture of cutter head of sugarcane harvester based on multi-sensor fusion | |
| US11596139B2 (en) | Method for autonomously weeding crops in an agricultural field | |
| US10172289B2 (en) | System and method for the automatic adjustment of the height of an agricultural implement using 3D reconstruction | |
| BR102014027364B1 (en) | cutting height measurement and control system for a basic harvester, method, and harvester | |
| CN201600330U (en) | System for recognizing and locating mature pineapples | |
| CN110235600A (en) | The Feeding Quantity in Combine stabilizing control system based on mature crop attribute information real-time detection | |
| CN113597874A (en) | Weeding robot and weeding path planning method, device and medium thereof | |
| CN110969654A (en) | Corn high-throughput phenotype measurement method and device based on harvester and harvester | |
| CN115861988B (en) | Tea picking method and system based on RGB (red, green and blue) distinction degree | |
| CN114667852B (en) | Hedge trimming robot intelligent cooperative control method based on deep reinforcement learning | |
| CN108196538A (en) | Field robots autonomous navigation system and method based on three-dimensional point cloud model | |
| CN115280964B (en) | Automatic operation driving method and system of stem and leaf vegetable harvester and harvester | |
| Bao et al. | Field-based robotic phenotyping for sorghum biomass yield component traits characterization using stereo vision | |
| CN115280960B (en) | Combined harvester steering control method based on field vision SLAM | |
| CN116117807A (en) | Chilli picking robot and control method | |
| CN112237089A (en) | Online prediction device and method for feeding amount of combine harvester | |
| CN107656287B (en) | A kind of Boundary Extraction device and method of the crudefiber crop row based on laser radar | |
| CN111867351B (en) | Method for determining the height of survival of a crop | |
| CN112837364A (en) | Cutting knife auxiliary positioning algorithm based on combination of binocular camera and monocular camera | |
| AU2023200747B2 (en) | Machine vision plant tracking system for precision agriculture | |
| Chimwai et al. | Fresh Weight Estimation of Lettuce Using 3D Stereoscopic Technique | |
| CN207992810U (en) | A kind of field automatic travelling device for detecting plant growth condition | |
| CN115268445A (en) | Robot autonomous targeting system and method with milemeter-crop canopy feature fusion | |
| CN116823812A (en) | Silage corn field life detection method | |
| CN117115769A (en) | Plant detection and positioning method based on semantic segmentation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |