CN117172123A - Sensor data processing method and system for mine automatic driving - Google Patents
Sensor data processing method and system for mine automatic driving Download PDFInfo
- Publication number
- CN117172123A CN117172123A CN202311179006.5A CN202311179006A CN117172123A CN 117172123 A CN117172123 A CN 117172123A CN 202311179006 A CN202311179006 A CN 202311179006A CN 117172123 A CN117172123 A CN 117172123A
- Authority
- CN
- China
- Prior art keywords
- bearing capacity
- training
- decision
- data
- road
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000012549 training Methods 0.000 claims abstract description 184
- 238000010801 machine learning Methods 0.000 claims abstract description 54
- 230000002787 reinforcement Effects 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000000034 method Methods 0.000 claims description 45
- 239000002689 soil Substances 0.000 claims description 32
- 239000011800 void material Substances 0.000 claims description 20
- 238000002474 experimental method Methods 0.000 claims description 17
- 230000005484 gravity Effects 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 9
- 238000013480 data collection Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000005065 mining Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 239000011435 rock Substances 0.000 description 2
- 239000002893 slag Substances 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Landscapes
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention discloses a sensor data processing method and a sensor data processing system for mine automatic driving, which relate to the technical field of mine automatic driving, and are characterized in that through collecting bearing capacity training data and line decision training data, a training four-element set is generated based on the line decision training data, a machine learning model for estimating the bearing capacity of a mine pavement is trained, and when a road cavity is detected, a deep reinforcement learning model for deciding whether to use an alternative driving route is trained; when the automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if the cavity exists, calculating the maximum bearing capacity of the road in the advancing direction, generating an alternative driving route, and outputting a decision of whether to use the alternative driving route based on the alternative driving route, the maximum bearing capacity and the deep reinforcement learning model; the safety of mine automatic driving is improved.
Description
Technical Field
The invention relates to the technical field of mine autopilot, in particular to a sensor data processing method and system for mine autopilot.
Background
Mines are used as important resource exploitation and production bases, and conventionally, materials transportation, ore collection and other works need to be carried out by means of manually driven vehicles. However, manual driving has a series of problems such as high labor intensity, severe working environment, high safety risk, and the like. In order to improve the transportation efficiency and the working safety of mines, mine autopilot technology has been developed.
The mine automatic driving technology is based on an advanced sensing, decision-making and control system, and utilizes sensors, navigation equipment, autonomous driving algorithms and the like to enable the vehicle to automatically sense the surrounding environment, plan the optimal path and conduct accurate autonomous navigation and operation. The automatic driving system can replace manual driving, and realizes autonomous running and operation of the mining vehicle;
however, in mines, geological conditions are complex and variable, and particularly, a hole may exist on the road surface of each area due to exploitation, so that collapse may occur during automatic driving of a vehicle through the hole. The existing automatic driving technology often lacks a method for deciding on the condition of a road hole in a mine;
an automatic surface mine driving method, electronic equipment and storage medium are disclosed in China patent with the application publication number of CN115862355A, and an automatic surface mine driving instruction is received, wherein the automatic surface mine driving instruction at least comprises an unmanned starting instruction and an unmanned task; switching the mining card driving mode into an unmanned mode according to the unmanned starting instruction; acquiring current coordinate position information of a corresponding mine card in a mining area; enumerating all preselected unmanned routes based on the current coordinate location information, the unmanned tasks, and the road layout of the mine area; according to the distance, road condition and traffic flow information of each pre-selected unmanned route, an optimal route is selected through a preset screening algorithm; enabling the corresponding ore cards to execute the unmanned task according to the optimal route; but this method fails to take into account the fact that there is a void under the road;
Therefore, the invention provides a sensor data processing method and system for mine automatic driving.
Disclosure of Invention
The present invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the invention provides a sensor data processing method and a system for mine automatic driving, which improve the safety of mine automatic driving.
To achieve the above object, embodiment 1 according to the present invention proposes a sensor data processing method for mine autopilot, comprising the steps of:
step one: the method comprises the steps that a server background collects bearing capacity training data and line decision training data;
step two: the server background generates a training four-element group set based on the line decision training data;
step three: the server background trains a machine learning model for estimating the bearing capacity of the mine pavement based on the bearing capacity training data, trains a deep reinforcement learning model for deciding whether to use an alternative driving route when a road hole is detected based on the training four-element set;
step four: loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background;
When an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if the cavity exists, turning to a step five;
step five: the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model;
step six: the automatic control background generates an alternative driving route and outputs a decision of whether to use the alternative driving route or not based on the alternative driving route, the maximum bearing capacity and the deep reinforcement learning model;
the bearing capacity training data comprise bearing capacity characteristic data and bearing capacity label data of each bearing capacity measurement experiment;
the bearing capacity characteristic data comprise surface thickness, soil type, soil moisture content, surface crack density and surface crack width;
the surface thickness is the distance between the surface of the mine road and the top of the underground cavity;
the soil types include, but are not limited to, slag soil, bare rock, virgin soil, and the like;
the soil moisture content can be obtained in real time through a soil humidity sensor;
The surface crack density is the average number of cracks of the unit area of the mine road surface;
the surface crack width is the width of the crack with the largest width in all cracks on the surface of the mine road;
the bearing capacity label data is the maximum bearing capacity of the mine road corresponding to each group of bearing capacity characteristic data; the maximum bearing capacity is a pressure value obtained by a pressure sensor when the load is continuously added on the mine road until the foundation collapses;
the line decision training data comprise test decision data of each automatic driving experiment;
the test decision data comprise starting point positions, end point positions and cavity decision training data of each automatic driving experiment;
when an automatic driving experiment is carried out, the tested automatic driving vehicle detects whether a hole exists in a road in the advancing direction by using a radar wave detection method, and the hole decision training data are data collected when the hole exists below the road in the advancing direction each time;
the cavity decision training data comprise bearing capacity label data of a road with a cavity, a cavity range, an alternative driving route, vehicle gravity, action decisions and decision results;
Wherein the void range is a range boundary of a void under a road with a void in the advancing direction;
the gravity of the vehicle is the pressure generated by the tested automatic driving vehicle on the road surface;
wherein the alternative driving route is an automatic driving route which bypasses the underground cavity;
the generation mode of the alternative driving route is as follows:
the method comprises the steps of obtaining a planned route from a starting point position to an end point position, which is generated in advance by navigation software, and calculating a starting intersection point and an ending intersection point of the planned route and a cavity range; the initial intersection point and the end intersection point are respectively the position of the planned route entering the cavity range and the position leaving the cavity range;
the distance of the alternative driving route is the shortest route from the initial intersection point to the end intersection point along the boundary line of the cavity range;
the action decision is selected when the tested automatic driving vehicle faces the cavity in each automatic driving experiment; the action of the selection is one of selecting or not selecting to use an alternative driving route;
the decision result is a road safety state after action decision is made; the road safety state is one of the road collapse where the automatic driving vehicle is and the road non-collapse;
The way of generating the training four-element group set based on the line decision training data is as follows:
generating a group of training quadruples for each group of cavity decision training data;
specifically, the way to generate a set of training quaternions is:
taking bearing capacity label data of each group of cavity decision training data, vehicle gravity and alternative driving route as the current state;
an action of taking an action decision of the hole decision training data as a selection action;
calculating a reward value Q of each group of cavity decision training data after the selection action;
the calculation mode of the reward value Q is as follows:
the method comprises the steps of marking the bearing capacity label data as C, marking the vehicle gravity as W, and marking the alternative driving route as L;
if the decision result is that the road collapses, the rewarding value Q is-Qmax; wherein Qmax is a preset maximum rewarding value;
if the decision result is that the road is not collapsed, the calculation formula of the rewarding value Q is as followsThe method comprises the steps of carrying out a first treatment on the surface of the Wherein x is one of 0 or 1, and x=0 when the action decision is to select to use the alternative travel route; when the action decision is not to select to use an alternative travel route, x=1; wherein a1 is a preset proportionality coefficient;
taking the decision result of the cavity decision training data as the next state;
The training quadruple comprises a current state, a selected action, a reward value Q and a next state which correspond to each group of cavity decision training data;
the training four-element set comprises training four elements of all cavity decision training data;
the mode of training a machine learning model for estimating the bearing capacity of the mine pavement is as follows:
taking each group of bearing capacity characteristic data as input of a machine learning model, wherein the machine learning model takes predicted bearing capacity label data of each group of bearing capacity characteristic data as output, takes bearing capacity label data corresponding to the bearing capacity characteristic data in bearing capacity training data as a prediction target, and takes the sum of prediction errors of all bearing capacity label data as a training target; training the machine learning model until the sum of the prediction errors reaches convergence, stopping training, and training the machine learning model which outputs predicted bearing capacity label data according to bearing capacity characteristic data; the machine learning model is one of a polynomial regression model or an SVM model;
the calculation formula of the prediction error is as follows:wherein g is the number of the bearing capacity characteristic data, xg is a prediction error, dg is predicted bearing capacity label data corresponding to the g-th group bearing capacity characteristic data, and eg is actual bearing capacity label data corresponding to the g-th group bearing capacity characteristic data;
The training method for deciding whether to use the deep reinforcement learning model of the alternative driving route when the road cavity is detected is as follows:
taking the current state and the rewarding value Q in the training four-element set as the input of a deep reinforcement learning model, wherein the deep reinforcement learning model carries out training by randomly extracting a plurality of four-element sets from the training four-element set, and learns whether to select to use an alternative driving route under different current states so as to obtain a strategy of the maximum rewarding value Q; the deep reinforcement learning model is a deep Q network model;
the method for automatically controlling the background to collect the bearing capacity characteristic data comprises the following steps:
the method comprises the steps of automatically controlling a background to acquire the monitored surface thickness in real time, collecting a road picture of the advancing direction through a vehicle-mounted camera of an automatic driving vehicle to be controlled, and analyzing and acquiring the soil type, the surface crack density and the surface crack width by using a computer vision technology, wherein the soil moisture content is acquired according to a plurality of soil moisture sensors which are installed in a mine in advance;
the mode of calculating the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and the machine learning model is as follows:
inputting the bearing capacity characteristic data collected by the automatic control background into a machine learning model to obtain output predicted bearing capacity label data as the maximum bearing capacity;
The way to output the decision whether to use the alternate travel route is:
and inputting the maximum bearing capacity obtained by the automatic control backstage, the generated alternative driving route and the gravity of the vehicle of the automatic driving vehicle to be controlled into a deep reinforcement learning model to obtain an output decision of whether to select to use the alternative driving route.
Embodiment 2 of the present invention proposes a sensor data processing system for mine autopilot, including a training data collection module, a model training module, and a route decision module; wherein, each module is connected by a wired mode;
the training data collection module is mainly used for collecting bearing capacity training data and line decision training data in a server background and generating a training quadruple set based on the line decision training data; the training data collection module sends bearing capacity training data and a training four-element set to the model training module;
the model training module is mainly used for training a machine learning model for estimating the bearing capacity of the mine pavement in a server background based on bearing capacity training data, training out a deep reinforcement learning model for deciding whether to use a substitute driving route when a road cavity is detected based on training four-element group set; the model training module sends the machine learning model and the deep reinforcement learning model which are completed in training to the route decision module;
The route decision module is mainly used for outputting a decision of whether to use an alternative driving route when an automatic control background of an automatic driving vehicle to be controlled monitors that a hole exists on a road in the advancing direction;
the way to output the decision whether to use the alternate travel route is:
loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background;
when an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if a cavity exists, the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model;
the automated control daemon generates an alternate travel route, and outputs a decision whether to use the alternate travel route based on the alternate travel route, the maximum load capacity, and the deep reinforcement learning model.
A computer server according to embodiment 3 of the present invention includes: a processor and a memory, wherein,
the memory stores a computer program which can be called by the processor;
The processor executes the sensor data processing method for mine autopilot by calling the computer program stored in the memory.
Embodiment 4 according to the present invention proposes a computer-readable storage medium having stored thereon a computer program that is erasable;
the computer program, when run on a computer device, causes the computer device to perform the sensor data processing method for mine autopilot described above.
Compared with the prior art, the invention has the beneficial effects that:
according to the method, the device and the system, the bearing capacity training data and the line decision training data are collected in the server background, a training quadruple set is generated based on the line decision training data, a machine learning model for estimating the bearing capacity of a mine pavement is trained based on the bearing capacity training data, a deep reinforcement learning model for deciding whether to use an alternative driving route is trained based on the training quadruple set when a road hole is detected, the server background loads the trained machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled, the automatic control background monitors whether a hole exists below a road in the advancing direction in real time in the process of controlling the automatic driving vehicle to be controlled according to a planned route, if the hole exists, the automatic control background collects bearing capacity characteristic data, calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and the machine learning model, and generates an alternative driving route based on the alternative driving route, the maximum bearing capacity and the deep reinforcement learning model, and outputs the decision whether to use the alternative driving route; whether a hole exists in a driving route of automatic driving in a mine or not is monitored in real time, and when the hole exists, whether an automatic driving vehicle to be controlled is controlled to bypass or not is intelligently decided, so that collapse caused by the hole of a road when the mine is automatically driven is avoided, and the safety of the automatic driving of the mine is improved.
Drawings
FIG. 1 is a flowchart of a sensor data processing method for mine autopilot in embodiment 1 of the present invention;
FIG. 2 is a block diagram of a sensor data processing system for mine autopilot in accordance with example 2 of the present invention;
fig. 3 is a schematic diagram of an electronic device according to embodiment 3 of the present invention;
fig. 4 is a schematic structural diagram of a computer readable storage medium according to embodiment 4 of the present invention.
Detailed Description
The technical solutions of the present invention will be clearly and completely described in connection with the embodiments, and it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
As shown in fig. 1, the sensor data processing method for mine autopilot comprises the following steps:
step one: the method comprises the steps that a server background collects bearing capacity training data and line decision training data;
step two: the server background generates a training four-element group set based on the line decision training data;
Step three: the server background trains a machine learning model for estimating the bearing capacity of the mine pavement based on the bearing capacity training data, trains a deep reinforcement learning model for deciding whether to use an alternative driving route when a road hole is detected based on the training four-element set;
step four: loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background;
when an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if the cavity exists, turning to a step five; if no cavity exists, continuing to monitor;
step five: the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model;
step six: the automatic control background generates an alternative driving route and outputs a decision of whether to use the alternative driving route or not based on the alternative driving route, the maximum bearing capacity and the deep reinforcement learning model;
the bearing capacity training data and the line decision training data are training data for model training, wherein the training data are generated by controlling a plurality of tested automatic driving vehicles to carry out bearing capacity measurement experiments and automatic driving experiments for a plurality of times in mines with different external environments and road environments;
The bearing capacity training data comprise bearing capacity characteristic data and bearing capacity label data of each bearing capacity measurement experiment;
the bearing capacity characteristic data comprise surface thickness, soil type, soil moisture content, surface crack density and surface crack width;
the surface thickness is the distance between the surface of the mine road and the top of the underground cavity; it can be appreciated that due to the mining of mines throughout the year, there may be many situations where voids exist below the road surface, resulting in a failure to carry the weight of the vehicle as it passes, and thus a risk of collapse; the earth surface thickness can be calculated according to the principle that the soil and the cavity have differences between the propagation speed and the reflection speed of the radar wave, and specifically, the China patent with the authority bulletin number of CN115220035B discloses a road cavity detection system and a road detection method, and provides a method for calculating the position depth of the cavity according to the propagation time of the radar wave and the transmission speed of the radar wave, which is not repeated herein;
the soil types include, but are not limited to, slag soil, bare rock, virgin soil, and the like; the soil type can be automatically identified through a computer vision technology, specifically, road pavement pictures of various soil types are collected in advance, each road pavement picture is marked with a label corresponding to the soil type, all the road pavement pictures are input into image identification neural network models such as a CNN neural network and the like for training, and the image identification neural network model for automatically identifying the soil type is obtained;
The soil moisture content can be obtained in real time through a soil humidity sensor; it can be understood that in the area range of the mine, the rainfall is small, and the depth of the underground water level is similar, so that a plurality of soil humidity sensors can be installed at a plurality of positions in the mine according to practical experience, and the average value of all the soil humidity sensors is calculated to be used as the soil moisture content;
the surface crack density is the average number of cracks of the unit area of the mine road surface; it can be understood that the cracks on the road surface can be identified and counted by a computer vision technology (R-CNN or Yolo3 model), and the average value of the number of cracks in each sub-picture is counted as the ground surface crack density by dividing the road picture into a plurality of sub-pictures according to the unit area;
the surface crack width is the width of the crack with the largest width in all cracks on the surface of the mine road; it will be appreciated that the crack width may be represented using the width of the crack region in the road picture, which may be identified using computer vision techniques;
the bearing capacity label data is the maximum bearing capacity of the mine road corresponding to each group of bearing capacity characteristic data; the maximum bearing capacity is a pressure value obtained by a pressure sensor when the load is continuously added on the mine road until the foundation collapses;
Further, the line decision training data comprises test decision data of each automatic driving experiment;
the test decision data comprise starting point positions, end point positions and cavity decision training data of each automatic driving experiment;
when an automatic driving experiment is carried out, the tested automatic driving vehicle detects whether a hole exists in a road in the advancing direction by using a radar wave detection method, and the hole decision training data are data collected when the hole exists below the road in the advancing direction each time;
the cavity decision training data comprise bearing capacity label data of a road with a cavity, a cavity range, an alternative driving route, vehicle gravity, action decisions and decision results;
wherein the void range is a range boundary of a void under a road with a void in the advancing direction; it should be noted that, the void range is obtained by extracting the boundary coordinates of the underground void, the Chinese patent with the authority bulletin number of CN109345592B discloses an underground void three-dimensional coordinate extraction algorithm based on ground penetrating radar, the coordinates of a plurality of boundary points of the underground void are obtained by adopting a singular value decomposition method, the coordinates of the boundary points are connected to generate a boundary line of the underground void, and the area surrounded by the boundary line is the void range; the present invention is not described in detail herein;
The gravity of the vehicle is the pressure generated by the tested automatic driving vehicle on the road surface; it can be understood that the gravity of the vehicle is calculated and obtained based on the mechanics principle through the weight of the vehicle and the inclination angle of the road surface;
wherein the alternative driving route is an automatic driving route which bypasses the underground cavity;
specifically, the generation mode of the alternative driving route is as follows:
the method comprises the steps of obtaining a planned route from a starting point position to an end point position, which is generated in advance by navigation software, and calculating a starting intersection point and an ending intersection point of the planned route and a cavity range; the initial intersection point and the end intersection point are respectively the position of the planned route entering the cavity range and the position leaving the cavity range; it can be understood that when the tested automatic driving vehicle detects that a cavity exists below the road in the advancing direction, the front of the vehicle is the initial intersection point;
the distance of the alternative driving route is the shortest route from the initial intersection point to the end intersection point along the boundary line of the cavity range; it will be appreciated that the clockwise detour and the anticlockwise detour can be selected from the start point to the end point along the boundary line, and therefore the shortest route is the route with the shortest path among the clockwise detour and the anticlockwise detour;
The action decision is selected when the tested automatic driving vehicle faces the cavity in each automatic driving experiment; the action of the selection is one of selecting or not selecting to use an alternative driving route;
the decision result is a road safety state after action decision is made; the road safety state is one of the road collapse where the automatic driving vehicle is and the road non-collapse;
the way of generating the training four-element group set based on the line decision training data is as follows:
generating a group of training quadruples for each group of cavity decision training data;
specifically, the way to generate a set of training quaternions is:
taking bearing capacity label data of each group of cavity decision training data, vehicle gravity and alternative driving route as the current state;
an action of taking an action decision of the hole decision training data as a selection action;
calculating a reward value Q of each group of cavity decision training data after the selection action;
the calculation mode of the reward value Q is as follows:
the method comprises the steps of marking the bearing capacity label data as C, marking the vehicle gravity as W, and marking the alternative driving route as L;
if the decision result is that the road collapses, the rewarding value Q is-Qmax; wherein Qmax is a preset maximum rewarding value;
If the decision result is that the road is not collapsed, the calculation formula of the rewarding value Q is as followsThe method comprises the steps of carrying out a first treatment on the surface of the Wherein x is one of 0 or 1, and x=0 when the action decision is to select to use the alternative travel route; when the action decision is not to select to use an alternative travel route, x=1; wherein a1 is a preset proportionality coefficient;
it will be appreciated that the greater the C-W, i.e. the greater the difference in force that the load bearing tag data can withstand from vehicle gravity, the more prone x=1, i.e. the more prone alternative travel routes are not selected for use; when L is smaller, i.eThe larger the distance of the detour, the smaller the distance of the detour, the more prone it is to make x=0, i.e. the alternative driving route is chosen to be used;
taking the decision result of the cavity decision training data as the next state;
the training quadruple comprises a current state, a selected action, a reward value Q and a next state which correspond to each group of cavity decision training data;
the training four-element set comprises training four elements of all cavity decision training data;
the mode of training a machine learning model for estimating the bearing capacity of the mine pavement is as follows:
taking each group of bearing capacity characteristic data as input of a machine learning model, wherein the machine learning model takes predicted bearing capacity label data of each group of bearing capacity characteristic data as output, takes bearing capacity label data corresponding to the bearing capacity characteristic data in bearing capacity training data as a prediction target, and takes the sum of prediction errors of all bearing capacity label data as a training target; training the machine learning model until the sum of the prediction errors reaches convergence, stopping training, and training the machine learning model which outputs predicted bearing capacity label data according to bearing capacity characteristic data; the machine learning model is one of a polynomial regression model or an SVM model;
The calculation formula of the prediction error is as follows:wherein g is the number of the bearing capacity characteristic data, xg is a prediction error, dg is predicted bearing capacity label data corresponding to the g-th group bearing capacity characteristic data, and eg is actual bearing capacity label data corresponding to the g-th group bearing capacity characteristic data;
the training method for deciding whether to use the deep reinforcement learning model of the alternative driving route when the road cavity is detected is as follows:
taking the current state and the rewarding value Q in the training four-element set as the input of a deep reinforcement learning model, wherein the deep reinforcement learning model carries out training by randomly extracting a plurality of four-element sets from the training four-element set, and learns whether to select to use an alternative driving route under different current states so as to obtain a strategy of the maximum rewarding value Q; the deep reinforcement learning model is a deep Q network model;
the way of automatically controlling the background to monitor whether the road in the advancing direction has a cavity in real time can be to install a vehicle-mounted radar on the body of an automatic driving vehicle to be controlled, then use the method of the Chinese patent with the authorized bulletin number of CN115220035B to monitor in real time, calculate the surface thickness and use the method of the Chinese patent with the authorized bulletin number of CN109345592B to obtain the cavity range;
The method for automatically controlling the background to collect the bearing capacity characteristic data comprises the following steps:
the method comprises the steps of automatically controlling a background to acquire the monitored surface thickness in real time, collecting a road picture of the advancing direction through a vehicle-mounted camera of an automatic driving vehicle to be controlled, and analyzing and acquiring the soil type, the surface crack density and the surface crack width by using a computer vision technology, wherein the soil moisture content is acquired according to a plurality of soil moisture sensors which are installed in a mine in advance;
the mode of calculating the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and the machine learning model is as follows:
inputting the bearing capacity characteristic data collected by the automatic control background into a machine learning model to obtain output predicted bearing capacity label data as the maximum bearing capacity;
the way to output the decision whether to use the alternate travel route is:
and inputting the maximum bearing capacity obtained by the automatic control backstage, the generated alternative driving route and the gravity of the vehicle of the automatic driving vehicle to be controlled into a deep reinforcement learning model to obtain an output decision of whether to select to use the alternative driving route.
Example 2
As shown in fig. 2, the sensor data processing system for mine autopilot comprises a training data collection module, a model training module and a route decision module; wherein, each module is connected by a wired mode;
The training data collection module is mainly used for collecting bearing capacity training data and line decision training data in a server background and generating a training quadruple set based on the line decision training data; the training data collection module sends bearing capacity training data and a training four-element set to the model training module;
the model training module is mainly used for training a machine learning model for estimating the bearing capacity of the mine pavement in a server background based on bearing capacity training data, training out a deep reinforcement learning model for deciding whether to use a substitute driving route when a road cavity is detected based on training four-element group set; the model training module sends the machine learning model and the deep reinforcement learning model which are completed in training to the route decision module;
the route decision module is mainly used for outputting a decision of whether to use an alternative driving route when an automatic control background of an automatic driving vehicle to be controlled monitors that a hole exists on a road in the advancing direction;
the way to output the decision whether to use the alternate travel route is:
loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background;
When an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if a cavity exists, the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model;
the automated control daemon generates an alternate travel route, and outputs a decision whether to use the alternate travel route based on the alternate travel route, the maximum load capacity, and the deep reinforcement learning model.
Example 3
Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 3, an electronic device 300 is also provided in accordance with yet another aspect of the present application. The electronic device 300 may include one or more processors and one or more memories. Wherein the memory has stored therein computer readable code which, when executed by the one or more processors, can perform the sensor data processing method for mine autopilot as described above.
The method according to an embodiment of the application may be implemented by means of the architecture of the electronic device shown in fig. 3. As shown in fig. 3, the electronic device 300 may include a bus 301, one or more CPUs 302, a Read Only Memory (ROM) 303, a Random Access Memory (RAM) 304, a communication port 305 connected to a network, an input/output component 306, a hard disk 307, and the like. A storage device in the electronic device 300, such as the ROM303 or the hard disk 307, may store the sensor data processing method for mine autopilot provided by the present application. Sensor data processing methods for mine autopilot may include, for example: step one: the method comprises the steps that a server background collects bearing capacity training data and line decision training data; step two: the server background generates a training four-element group set based on the line decision training data; step three: the server background trains a machine learning model for estimating the bearing capacity of the mine pavement based on the bearing capacity training data, trains a deep reinforcement learning model for deciding whether to use an alternative driving route when a road hole is detected based on the training four-element set; step four: loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background; when an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if the cavity exists, turning to a step five; if no cavity exists, continuing to monitor; step five: the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model; step six: the automated control daemon generates an alternate travel route, and outputs a decision whether to use the alternate travel route based on the alternate travel route, the maximum load capacity, and the deep reinforcement learning model. Further, the electronic device 300 may also include a user interface 308. Of course, the architecture shown in fig. 3 is merely exemplary, and one or more components of the electronic device shown in fig. 3 may be omitted as may be practical in implementing different devices.
Example 4
FIG. 4 is a schematic diagram of a computer-readable storage medium according to one embodiment of the present application. As shown in fig. 4, is a computer-readable storage medium 400 according to one embodiment of the application. Computer readable storage medium 400 has stored thereon computer readable instructions. The sensor data processing method for mine autopilot according to the embodiment of the present application described with reference to the above drawings may be performed when the computer readable instructions are executed by the processor. Computer-readable storage medium 400 includes, for example, but is not limited to, volatile memory and/or nonvolatile memory. Volatile memory can include, for example, random Access Memory (RAM), cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like.
In addition, according to embodiments of the present application, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, the present application provides a non-transitory machine-readable storage medium storing machine-readable instructions executable by a processor to perform instructions corresponding to the method steps provided by the present application, such as: step one: the method comprises the steps that a server background collects bearing capacity training data and line decision training data; step two: the server background generates a training four-element group set based on the line decision training data; step three: the server background trains a machine learning model for estimating the bearing capacity of the mine pavement based on the bearing capacity training data, trains a deep reinforcement learning model for deciding whether to use an alternative driving route when a road hole is detected based on the training four-element set; step four: loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background; when an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if the cavity exists, turning to a step five; if no cavity exists, continuing to monitor; step five: the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model; step six: the automated control daemon generates an alternate travel route, and outputs a decision whether to use the alternate travel route based on the alternate travel route, the maximum load capacity, and the deep reinforcement learning model. The above-described functions defined in the method of the present application are performed when the computer program is executed by a Central Processing Unit (CPU).
The methods and apparatus, devices of the present application may be implemented in numerous ways. For example, the methods and apparatus, devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above-described sequence of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the sequence specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present application may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present application. Thus, the present application also covers a recording medium storing a program for executing the method according to the present application.
In addition, in the foregoing technical solutions provided in the embodiments of the present application, parts consistent with implementation principles of corresponding technical solutions in the prior art are not described in detail, so that redundant descriptions are avoided.
The above preset parameters or preset thresholds are set by those skilled in the art according to actual conditions or are obtained by mass data simulation.
The above embodiments are only for illustrating the technical method of the present application and not for limiting the same, and it should be understood by those skilled in the art that the technical method of the present application may be modified or substituted without departing from the spirit and scope of the technical method of the present application.
Claims (14)
1. Sensor data processing method for mine autopilot, characterized by comprising the steps of:
step one: the method comprises the steps that a server background collects bearing capacity training data and line decision training data;
step two: the server background generates a training four-element group set based on the line decision training data;
step three: the server background trains a machine learning model for estimating the bearing capacity of the mine pavement based on the bearing capacity training data, trains a deep reinforcement learning model for deciding whether to use an alternative driving route when a road hole is detected based on the training four-element set;
step four: loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background; when an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if the cavity exists, turning to a step five;
step five: the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model;
step six: the automated control daemon generates an alternate travel route, and outputs a decision whether to use the alternate travel route based on the alternate travel route, the maximum load capacity, and the deep reinforcement learning model.
2. The sensor data processing method for mine autopilot of claim 1 wherein the load bearing training data includes load bearing characteristic data for each load bearing measurement experiment and load bearing label data;
the bearing capacity characteristic data comprise surface thickness, soil type, soil moisture content, surface crack density and surface crack width;
the surface thickness is the distance between the surface of the mine road and the top of the underground cavity;
the soil moisture content is obtained in real time through a soil humidity sensor;
the surface crack density is the average number of cracks of the unit area of the mine road surface;
the surface crack width is the width of the crack with the largest width in all cracks on the surface of the mine road;
the bearing capacity label data is the maximum bearing capacity of the mine road corresponding to each group of bearing capacity characteristic data; the maximum bearing capacity is a pressure value obtained by a pressure sensor when the load is continuously added on the mine road until the foundation collapses.
3. The sensor data processing method for mine autopilot of claim 2 wherein the line decision training data includes test decision data for each autopilot experiment;
The test decision data comprise starting point positions, end point positions and cavity decision training data of each automatic driving experiment;
when an automatic driving experiment is carried out, the tested automatic driving vehicle detects whether a hole exists in a road in the advancing direction by using a radar wave detection method, and the hole decision training data are data collected when the hole exists below the road in the advancing direction each time;
the void decision training data comprise bearing capacity label data of a road with a void, a void range, an alternative driving route, vehicle gravity, action decisions and decision results.
4. A sensor data processing method for mine autopilot as claimed in claim 3 wherein the void range is a range boundary of a void under a road in the forward direction where the void exists;
the gravity of the vehicle is the pressure generated by the tested automatic driving vehicle on the road surface;
the alternative driving route is an automatic driving route which bypasses the underground cavity;
the action decision is selected when the tested automatic driving vehicle faces the cavity in each automatic driving experiment; the action of the selection is one of selecting or not selecting to use an alternative driving route;
The decision result is a road safety state after action decision is made; the road safety state is one of the collapse and the non-collapse of the road where the automatic driving vehicle is located.
5. The sensor data processing method for mine autopilot of claim 4 wherein the alternate travel route is generated by:
the method comprises the steps of obtaining a planned route from a starting point position to an end point position, which is generated in advance by navigation software, and calculating a starting intersection point and an ending intersection point of the planned route and a cavity range; the initial intersection point and the end intersection point are respectively the position of the planned route entering the cavity range and the position leaving the cavity range;
the alternate travel route is the shortest route from the start intersection point to the end intersection point along the boundary line of the cavity range.
6. The method for sensor data processing for mine autopilot of claim 5 wherein the means for generating a training quadruple set based on the line decision training data is:
generating a group of training quadruples for each group of cavity decision training data;
the way to generate a set of training quaternions is:
taking bearing capacity label data of each group of cavity decision training data, vehicle gravity and alternative driving route as the current state;
An action of taking an action decision of the hole decision training data as a selection action;
calculating a reward value Q of each group of cavity decision training data after the selection action;
taking the decision result of the cavity decision training data as the next state;
the training quadruple comprises a current state, a selected action, a reward value Q and a next state which correspond to each group of cavity decision training data;
the training quadruple set comprises training quadruples of all cavity decision training data.
7. The sensor data processing method for mine autopilot of claim 6 wherein the prize value Q is calculated by:
the method comprises the steps of marking the bearing capacity label data as C, marking the vehicle gravity as W, and marking the alternative driving route as L;
if the decision result is that the road collapses, the rewarding value Q is-Qmax; wherein Qmax is a preset maximum rewarding value;
if the decision result is that the road is not collapsed, the calculation formula of the rewarding value Q is as followsThe method comprises the steps of carrying out a first treatment on the surface of the Wherein x is one of 0 or 1, and x=0 when the action decision is to select to use the alternative travel route; when the action decision is not to select to use an alternative travel route, x=1; wherein a1 is a preset proportionality coefficient.
8. The method for processing sensor data for mine autopilot of claim 7 wherein the machine learning model for estimating mine pavement load bearing capacity is trained in the following manner:
taking each group of bearing capacity characteristic data as input of a machine learning model, wherein the machine learning model takes predicted bearing capacity label data of each group of bearing capacity characteristic data as output, takes bearing capacity label data corresponding to the bearing capacity characteristic data in bearing capacity training data as a prediction target, and takes the sum of prediction errors of all bearing capacity label data as a training target; training the machine learning model until the sum of the prediction errors reaches convergence, stopping training, and training the machine learning model which outputs predicted bearing capacity label data according to bearing capacity characteristic data;
the calculation formula of the prediction error is as follows:wherein g is the number of the bearing capacity characteristic data, xg is the prediction error, dg is the predicted bearing capacity label data corresponding to the g-th group of bearing capacity characteristic data, and eg is the actual bearing capacity label data corresponding to the g-th group of bearing capacity characteristic data.
9. The sensor data processing method for mine autopilot of claim 8 wherein the training to decide whether to use a deep reinforcement learning model of an alternate travel route upon detection of a road void is:
And taking the current state and the rewarding value Q in the training four-element set as the input of a deep reinforcement learning model, wherein the deep reinforcement learning model carries out training by randomly extracting a plurality of four-elements from the training four-element set, and learns whether to select to use an alternative driving route under different current states so as to obtain a strategy of the maximum rewarding value Q.
10. The sensor data processing method for mine autopilot of claim 9 wherein the way to calculate the maximum bearing capacity of the road in the forward direction based on the bearing capacity feature data and the machine learning model is:
and inputting the bearing capacity characteristic data collected by the automatic control background into a machine learning model to obtain output predicted bearing capacity label data as the maximum bearing capacity.
11. The sensor data processing method for mine autopilot of claim 10 wherein the way to output a decision whether to use an alternate travel route is:
and inputting the maximum bearing capacity obtained by the automatic control backstage, the generated alternative driving route and the gravity of the vehicle of the automatic driving vehicle to be controlled into a deep reinforcement learning model to obtain an output decision of whether to select to use the alternative driving route.
12. Sensor data processing system for mine autopilot, realized on the basis of the sensor data processing method for mine autopilot according to any one of claims 1-11, characterized by comprising a training data collection module, a model training module and a route decision module; wherein, each module is connected by a wired mode;
the training data collection module is used for collecting bearing capacity training data and line decision training data in the background of the server and generating a training quadruple set based on the line decision training data; the training data collection module sends bearing capacity training data and a training four-element set to the model training module;
the model training module trains a machine learning model for estimating the bearing capacity of the mine pavement on the basis of bearing capacity training data in a server background, trains a deep reinforcement learning model for deciding whether to use a substitute driving route when a road cavity is detected on the basis of training four-element group set; the model training module sends the machine learning model and the deep reinforcement learning model which are completed in training to the route decision module;
the route decision module outputs a decision of whether to use an alternative driving route when an automatic control background of an automatic driving vehicle to be controlled monitors that a hole exists in a road in the advancing direction;
The way to output the decision whether to use the alternate travel route is:
loading the machine learning model and the deep reinforcement learning model into an automatic control background of an automatic driving vehicle to be controlled by a server background;
when an automatic driving vehicle to be controlled automatically drives according to the planned route data, the automatic control background monitors whether a hole exists on a road in the advancing direction in real time; if a cavity exists, the automatic control background collects bearing capacity characteristic data, and calculates the maximum bearing capacity of the road in the advancing direction based on the bearing capacity characteristic data and a machine learning model;
the automated control daemon generates an alternate travel route, and outputs a decision whether to use the alternate travel route based on the alternate travel route, the maximum load capacity, and the deep reinforcement learning model.
13. A computer server, comprising: a processor and a memory, wherein the memory stores a computer program for the processor to call;
the processor performs the sensor data processing method for mine autopilot of any one of claims 1 to 11 by invoking a computer program stored in the memory.
14. A computer readable storage medium, characterized in that it stores a computer program that is erasable;
The computer program, when run on a computer device, causes the computer device to perform the sensor data processing method for mine autopilot of any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311179006.5A CN117172123B (en) | 2023-09-13 | 2023-09-13 | Sensor data processing method and system for mine automatic driving |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311179006.5A CN117172123B (en) | 2023-09-13 | 2023-09-13 | Sensor data processing method and system for mine automatic driving |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117172123A true CN117172123A (en) | 2023-12-05 |
| CN117172123B CN117172123B (en) | 2024-03-08 |
Family
ID=88940994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311179006.5A Active CN117172123B (en) | 2023-09-13 | 2023-09-13 | Sensor data processing method and system for mine automatic driving |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117172123B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160132705A1 (en) * | 2014-11-12 | 2016-05-12 | Joseph E. Kovarik | Method and System for Autonomous Vehicles |
| US20190225234A1 (en) * | 2019-03-29 | 2019-07-25 | Intel Corporation | Detection of an anomalous image associated with image data from one or more cameras of a computer-aided or autonomous driving vehicle |
| US20200307577A1 (en) * | 2019-03-26 | 2020-10-01 | Ford Global Technologies, Llc | Interpreting data of reinforcement learning agent controller |
| CN113561986A (en) * | 2021-08-18 | 2021-10-29 | 武汉理工大学 | Decision-making method and device for automatically driving automobile |
| CN114348021A (en) * | 2022-01-05 | 2022-04-15 | 清华大学 | Vehicle autonomous extreme driving planning control method and system based on reinforcement learning |
| CN116476863A (en) * | 2023-03-30 | 2023-07-25 | 合肥工业大学 | Automatic driving transverse and longitudinal integrated decision-making method based on deep reinforcement learning |
| CN116653962A (en) * | 2023-07-12 | 2023-08-29 | 南京项尚车联网技术有限公司 | Intelligent driving vehicle efficient energy-saving system and energy-saving method |
-
2023
- 2023-09-13 CN CN202311179006.5A patent/CN117172123B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160132705A1 (en) * | 2014-11-12 | 2016-05-12 | Joseph E. Kovarik | Method and System for Autonomous Vehicles |
| US20200307577A1 (en) * | 2019-03-26 | 2020-10-01 | Ford Global Technologies, Llc | Interpreting data of reinforcement learning agent controller |
| US20190225234A1 (en) * | 2019-03-29 | 2019-07-25 | Intel Corporation | Detection of an anomalous image associated with image data from one or more cameras of a computer-aided or autonomous driving vehicle |
| CN113561986A (en) * | 2021-08-18 | 2021-10-29 | 武汉理工大学 | Decision-making method and device for automatically driving automobile |
| CN114348021A (en) * | 2022-01-05 | 2022-04-15 | 清华大学 | Vehicle autonomous extreme driving planning control method and system based on reinforcement learning |
| CN116476863A (en) * | 2023-03-30 | 2023-07-25 | 合肥工业大学 | Automatic driving transverse and longitudinal integrated decision-making method based on deep reinforcement learning |
| CN116653962A (en) * | 2023-07-12 | 2023-08-29 | 南京项尚车联网技术有限公司 | Intelligent driving vehicle efficient energy-saving system and energy-saving method |
Non-Patent Citations (1)
| Title |
|---|
| 夏伟;李慧云;: "基于深度强化学习的自动驾驶策略学习方法", 集成技术, no. 03, 15 May 2017 (2017-05-15), pages 29 - 35 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117172123B (en) | 2024-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6764842B2 (en) | Information processing equipment, information processing system, information processing method, and information processing program | |
| EP2283315B1 (en) | Method and arrangement for calculating a conformity between a representation of an environment and said environment | |
| CN110674705B (en) | Small-sized obstacle detection method and device based on multi-line laser radar | |
| KR20200100810A (en) | Own vehicle location prediction method and autonomous driving device for autonomous driving | |
| CN110866433A (en) | Method and device for determining a road marking | |
| CN105426858A (en) | Vision and vibration information fusion based ground type identification method | |
| RU2017101247A (en) | COMPARATIVE ANALYSIS ON THE MODEL OF DRIFT AND TOWING ICE FOR TARGET MARINE CONSTRUCTION | |
| KR102576001B1 (en) | Automation device and method for evaluating Rock Mass Rating(RMR) using deep learning models | |
| CN117765708A (en) | Slope instability prediction method and device based on map image analysis technology | |
| WO2021241487A1 (en) | Timing prediction method, timing prediction device, timing prediction system, program, and construction machinery system | |
| CN115236997B (en) | Predictive control method, apparatus, device, and computer-readable storage medium | |
| CN113383283A (en) | Perception information processing method and device, computer equipment and storage medium | |
| CN117172123B (en) | Sensor data processing method and system for mine automatic driving | |
| CN111488280B (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN116295497A (en) | Path generation method, path generation device, robot and storage medium | |
| KR102519715B1 (en) | Road information providing system and method | |
| CN115683142A (en) | Method and device for determining region of interest | |
| CN115438945A (en) | Risk identification method, device, equipment and medium based on power equipment inspection | |
| KR101706455B1 (en) | Road sign detection-based driving lane estimation method and apparatus | |
| JP2019057221A (en) | Information processor, information processing system, method for processing information, and information processing program | |
| CN113762030A (en) | Data processing method and device, computer equipment and storage medium | |
| CN108459996A (en) | A kind of user's stroke method for digging and device based on supervised learning | |
| CN113609947A (en) | Motion trajectory prediction method, motion trajectory prediction device, computer equipment and storage medium | |
| CN113361317A (en) | System and method for end-to-end prediction of lane detection uncertainty | |
| CN113741448B (en) | Full-automatic control method and device for Mars vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |