CN110514225A - The calibrating external parameters and precise positioning method of Multi-sensor Fusion under a kind of mine - Google Patents

The calibrating external parameters and precise positioning method of Multi-sensor Fusion under a kind of mine Download PDF

Info

Publication number
CN110514225A
CN110514225A CN201910806408.0A CN201910806408A CN110514225A CN 110514225 A CN110514225 A CN 110514225A CN 201910806408 A CN201910806408 A CN 201910806408A CN 110514225 A CN110514225 A CN 110514225A
Authority
CN
China
Prior art keywords
coordinate system
under
uwb
measurement unit
inertial measurement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910806408.0A
Other languages
Chinese (zh)
Other versions
CN110514225B (en
Inventor
李猛钢
朱华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China University of Mining and Technology CUMT
Original Assignee
China University of Mining and Technology CUMT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China University of Mining and Technology CUMT filed Critical China University of Mining and Technology CUMT
Priority to CN201910806408.0A priority Critical patent/CN110514225B/en
Publication of CN110514225A publication Critical patent/CN110514225A/en
Application granted granted Critical
Publication of CN110514225B publication Critical patent/CN110514225B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C25/00Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
    • G01C25/005Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass initial alignment, calibration or starting-up of inertial devices
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01SRADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S5/00Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
    • G01S5/02Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using radio waves
    • G01S5/0205Details
    • G01S5/021Calibration, monitoring or correction
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01SRADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/497Means for monitoring or calibrating

Abstract

The invention discloses a kind of calibrating external parameters of Multi-sensor Fusion under mine and precise positioning method, the step of calibrating external parameters: obtaining absolute position of each UWB anchor node under world coordinate system in roadway, obtain the observation information of each sensor;It determines and joins outside the rotation between laser radar sensor and Inertial Measurement Unit, and judge automatically whether rotation calibration process is completed;The linear estimator based on sliding window is constructed, estimates correlated condition amount, and judge automatically whether linear estimator restrains;It using position coordinates of the UWB anchor node estimated under local coordinate system, is aligned with absolute location coordinates of the UWB anchor node under world coordinate system, determines the outer ginseng transformation between local coordinate system and world coordinate system;Export the parameter value of all estimations.The present invention can be realized online real-time or intermittent external parameter automatic Calibration, and precisely construct map and positioning.

Description

The calibrating external parameters and precise positioning method of Multi-sensor Fusion under a kind of mine
Technical field
The invention belongs to Mine pit survey fields, in particular to the sensor external parameter calibration method under a kind of mine.
Background technique
Special vehicle or robot, the especially intelligence for having autonomous ability in the work of closed underground coal mine Mining robot needs the result based on Multi-sensor Fusion to realize reliable whole scene seamless positioning and map structuring.It is right In the robot of underground work, laser radar (LiDAR)/Inertial Measurement Unit (IMU)/ultra wide band mould group multisensor is used Fusion method is effective scheme.Laser radar has the characteristics that measurement accuracy is high, scanning range is wide, strong signal-to-noise ratio, the disadvantage is that Under the similar tunnel structure degeneration scene of structure, matching is scanned using laser point cloud and feature extraction is all highly difficult, is led Cause positioning accuracy low.The short-term measurement accuracy of Inertial Measurement Unit is high, is suitble to jolt, under the occasions such as rotation is quickly other sensors Short-term reliable state estimation is provided, but directly accumulated error is very big under severe conditions using inertial navigation.Ultra-wide The range accuracy of band is high, has better reliability compared to other wireless location modes, for multipath effect and dropout With stronger rejection ability, the disadvantage is that directly carrying out needing when two and three dimensions positioning using a large amount of fixed base stations, basis is set It is larger to apply deployment cost, but can be used as aiding sensors and carry out accurate one-dimensional positioning in roadway direction to reduce cost. Mining robot may be implemented using laser radar/Inertial Measurement Unit/ultra wide band mould group multiple sensor integrated method to seal It closes, accurate positioning and high-precision environment modeling under severe complex scene, successfully manages degeneration, bumpy road when scene lacks structure The possible operating conditions such as vibration is violent, wireless signal is lost, while the usage quantity of UWB fixed base stations is reduced, reduce cost.
The premise of Multi-sensor Fusion is outer ginseng transformation relation between accurately known sensor (including between sensor Relative translation and relative rotation), to realize unified state estimation.Due to work long hours jolt in underground, vibrate acutely, Under the conditions of the environment work frequently hit, the outer ginseng between sensor inevitably changes, and needs to have On-line Estimation Method come in real time or intermittent carry out outer ginseng calibration and automatic calibration.Either academia or industry, outer ginseng are marked at present Surely it is concentrated mainly on laser radar/camera, it, can on-line proving simultaneously without special method on camera/Inertial Measurement Unit Laser radar/Inertial Measurement Unit/ultra wide band mould group.
Therefore, it is led to meet pit robot and other work in the case where closing scene using three-dimensional laser radar/inertia Boat unit/ultra wide band mould group is positioned, is built the actual demand of the tasks such as figure, perception, it would be highly desirable to which designing can be with automatic on-line mark The method of its fixed external parameter (hereinafter referred to as outer ginseng).
Summary of the invention
In order to solve the technical issues of above-mentioned background technique is mentioned, the invention proposes Multi-sensor Fusions under a kind of mine Calibrating external parameters and precise positioning method.
In order to achieve the above technical purposes, the technical solution of the present invention is as follows:
The method for calibrating external parameters of Multi-sensor Fusion under a kind of mine, the multisensor include laser radar sensing Device, Inertial Measurement Unit and ultra wide band mould group, the ultra wide band mould group include UWB mobile node and several UWB anchor nodes, described Laser radar sensor, Inertial Measurement Unit and UWB mobile node are separately positioned on underground mobile equipment, several UWB Anchor node is distributed in different location in roadway;The method for calibrating external parameters is as follows:
(1) absolute position of each UWB anchor node under world coordinate system in roadway is obtained, the observation of each sensor is obtained Information;
(2) control underground mobile equipment carries out the movement comprising centainly rotating, and determines that laser radar sensor and inertia are surveyed It measures and joins outside the rotation between unit, and judge automatically whether rotation calibration process is completed;
(3) linear estimator based on sliding window is constructed, the direction for estimating gravity, each UWB anchor node are in local coordinate Ginseng, UWB mobile node and inertia measurement list outside position, laser radar sensor under system and the translation between Inertial Measurement Unit Join outside translation between member, position and speed of the Inertial Measurement Unit under local coordinate system, and judges automatically linear estimator Whether restrain;
(4) position coordinates using the UWB anchor node estimated under local coordinate system are sat with UWB anchor node in the world Absolute location coordinates under mark system are aligned, and determine the outer ginseng transformation between local coordinate system and world coordinate system;
(5) parameter value of all estimations under world coordinate system is exported.
Further, in step (1), absolute position of each UWB anchor node under world coordinate system in roadway is obtained Method includes: to be obtained using underground gis database, obtained by field surveys and utilize measuring instrument from roadway In lead forever a little and elevational point carry out connection survey acquisition.
Further, in step (2), the rotation between laser radar sensor and Inertial Measurement Unit is determined by following formula Turn outer ginseng:
Wherein,For the amount of relative rotation of adjacent time inter Inertial Measurement Unit;For laser radar to be estimated Join outside rotation between sensor and Inertial Measurement Unit;For the relative rotation of adjacent time inter laser radar sensor Amount.
The method for judging whether rotation calibration process is completed is as follows:
Collect observation of the N to the Inertial Measurement Unit in the observation of laser radar sensor and corresponding time interval, construction Overdetermined equation:
Wherein:
Wherein,ForQuaternary number expression, QNFor overdetermination matrix, λiThe noise and outer dot factor of i-th pair observation are characterized,WithIt is respectivelyWithIt is first three element of quaternary number qxyzAntisymmetric matrix, qwIt is the of quaternary number Four elements, Ι3For 3 × 3 unit matrix;
Calculate overdetermination matrix QNThe second small singular value σmin2, with preset threshold value σthresholdIt is compared, such as Fruit meets σmin2> σthreshold, then overdetermined equation has solution, terminates rotation calibration process.
Further, in step (4), UTM is converted by absolute coordinate of each UWB anchor node under world coordinate system Coordinate under coordinate systemThe position estimation value P of each UWB anchor node under local coordinate system is obtained simultaneouslyi, construct error equation:
Wherein, M is that available UWB observes quantity in current sliding window,For the outer ginseng of local coordinate system and world coordinate system Transformation;
The M+1 position coordinates newly observed are added to above-mentioned error equation, obtain eM+1, utilize the condition of convergenceThe outer ginseng transformation of iterative solutionWherein, Δ is normalization variable, and ε is preset threshold.
Further, in step (3), corresponding inertia is observed with first frame laser radar sensor in sliding window and is surveyed The pose for measuring unit constructs local coordinate system as starting point, determines that used all the sensors are observed using sliding window, line Property estimator observe the sum of geneva norm of residual error by minimizing all the sensors and determine variable to be estimated:
Wherein, χ is variable to be estimated:
Wherein, xn,xn+1,…,xn+NIndicate Inertial Measurement Unit state, includePositionWith the direction of gravitySubscript n refers to Inertial Measurement Unit, and subscript N is the number of states of Inertial Measurement Unit in current sliding window mouth;For laser Join outside translation between radar sensor and Inertial Measurement Unit;It is flat between UWB mobile node and Inertial Measurement Unit Move outer ginseng;pm,pm+1,…,pm+MIndicate position of the UWB anchor node under local coordinate system, subscript m refers to UWB observation, subscript M It is all available UWB observation quantity in current window;rPFor the priori factor residual error item after marginalisation, HpIt is first after marginalisation Test the Jacobian matrix of factor residual error item;Residual error item is observed for Inertial Measurement Unit, is carried out using pre-integration method Building,For Inertial Measurement Unit observational equation,For corresponding covariance matrix, B is inertia all in sliding window Measuring unit pre-integration observes quantity;Residual error item is observed for laser radar sensor, passes through fitting point of proximity to subgraph In plane, constructed using distance of the point to face as matching measurement,For laser radar sensor observational equation, PsFor Corresponding covariance matrix, A are all quantity that matched radar observation is carried out with subgraph;Residual error is observed for UWB , building is observed using the distance of UWB mobile node to UWB anchor node,For UWB observational equation,For corresponding covariance Matrix, C are that all UWB observe quantity in sliding window;
Solve linear equation:
PBLU) χ=(bP+bB+bL+bU)
Wherein, ΛB,bBIt is the information matrix and vector of Inertial Measurement Unit observation;ΛL,bLIt is that laser radar sensor is seen The information matrix and vector of survey;ΛU,bUIt is the information matrix and vector of UWB observation;Λp,bpIt is the information matrix of priori factor And vector;
Judge that whether convergent linear estimator method be as follows:
Calculate (ΛPBLU)-1Maximum singular value λmax, by itself and preset threshold λthresholdIt is compared, if λmax< λthresholdThen judge that linear estimator is restrained.
Based on the precise positioning method of the method for calibrating external parameters of Multi-sensor Fusion under above-mentioned mine, including following step It is rapid:
(a) using the outer ginseng estimated result of the method for calibrating external parameters output of Multi-sensor Fusion under mine as initial value, Construct the Continuous optimization estimation that nonlinear optimization estimator carries out calibration result;
(b) pose using the underground mobile equipment of estimation under local coordinate system and local coordinate system and the world are sat The outer ginseng transformation relation for marking system, determines pose of the underground mobile equipment under world coordinate system;
(c) pose of the underground mobile equipment under world coordinate system, and the laser radar sensor point cloud obtained are utilized Information is registered under world coordinate system, obtains global map;
(d) estimated using the pose of global map and the laser radar sensor under world coordinate system, to posture information It is integrated, obtains the accurate positioning result under global map;
(e) result will be accurately positioned and returns to nonlinear optimization estimator, start to estimate next time.
Further, which is characterized in that in step (a), the state variable of the nonlinear optimization estimator is using mistake Poor state indicates:
Wherein, δ indicates error symbol;
Nonlinear optimization estimator is estimated by minimizing the sum of the geneva norm of all observation residual errors to obtain maximum a posteriori Meter:
Wherein,For the Jacobian matrix of Inertial Measurement Unit observation, HsFor laser radar sensor observation Jacobian matrix,For the Jacobian matrix of UWB observation;
Solve nonlinear equation:
PBLU) δ χ=(bP+bB+bL+bU)
Above-mentioned nonlinear equation, error state renewal process are iteratively solved using nonlinear optimization estimator are as follows:WithFor the state variable that front and back iteratively solves twice,Operation is for the variable of theorem in Euclid space Simple add operation is the multiplying of quaternary number for rotation process.
Further, it in step (b), is integrated by the matrix of matrix multiplication and operates determining underground mobile equipment in the world Pose under coordinate system
Wherein,For pose of the underground mobile equipment estimated by nonlinear estimator under local coordinate system,To estimate The local coordinate system of meter and the outer ginseng transformation relation of world coordinate system.
Further, in step (c), during registering point cloud information to world coordinate system, using underground moving Pose of the equipment under world coordinate system obtains the current point cloud information of distortionless laser radar sensor as movement priori Sk, it is registered to existing global map under world coordinate systemIn, obtain new global map
Further, in step (d), global map is utilizedAnd it is transformed into the laser radar under world coordinate system Sensor point cloud information Sk+1, utilize Sk+1It arrivesScan matching, calculate obtain build figure process pose transformationTo building figure The pose transformation that process obtainsWith posture information of the underground mobile equipment under world coordinate systemIt is integrated, is obtained complete Accurate positioning result under local figure
By adopting the above technical scheme bring the utility model has the advantages that
The present invention realizes online real-time or intermittent external parameter automatic Calibration, together by design linear estimator When design nonlinear optimization estimator, duration optimization is carried out to the external parameter of linear estimator output, and according to non-linear The result of optimal estimating device output carries out accurate map structuring and positioning.The present invention without rely on specific mechanical arrangements or Special Laboratory Calibration place is suitble to the use of the inferior complex industrial working scene of coal mine.
Detailed description of the invention
Fig. 1 is method for calibrating external parameters flow chart of the present invention;
Fig. 2 is precise positioning method flow diagram of the present invention.
Specific embodiment
Below with reference to attached drawing, technical solution of the present invention is described in detail.
The present invention devises a kind of method for calibrating external parameters of Multi-sensor Fusion under mine, and the multisensor includes Laser radar sensor, Inertial Measurement Unit and ultra wide band mould group, the ultra wide band mould group include UWB mobile node and several UWB anchor node, the laser radar sensor, Inertial Measurement Unit and UWB mobile node are separately positioned on underground mobile equipment On, several UWB anchor nodes are distributed in different location in roadway.As shown in Figure 1, the method for calibrating external parameters is as follows:
Step 101: obtaining absolute position of each UWB anchor node under world coordinate system in roadway, obtain each sensor Observation information.
Step 102: control underground mobile equipment carries out the movement comprising centainly rotating, and determines laser radar sensor and is used to Property measuring unit between rotation outside join, and judge automatically rotation calibration process whether complete.
Step 103: linear estimator of the building based on sliding window, the direction for estimating gravity, each UWB anchor node are in part Join outside position, laser radar sensor under coordinate system and the translation between Inertial Measurement Unit, UWB mobile node and inertia are surveyed Ginseng, position and speed of the Inertial Measurement Unit under local coordinate system outside the translation between unit are measured, and judges automatically and linearly estimates Whether gauge restrains.
Step 104: alive with UWB anchor node using position coordinates of the UWB anchor node estimated under local coordinate system Absolute location coordinates under boundary's coordinate system are aligned, and determine the outer ginseng transformation between local coordinate system and world coordinate system.
Step 105: the parameter value of all estimations under output world coordinate system.
In the present embodiment, above-mentioned steps 101 can use following preferred embodiment:
The method for obtaining absolute position of each UWB anchor node under world coordinate system in roadway includes: to utilize underground geography Information system database obtains, is obtained by field surveys and utilization measuring instrument (such as total station, theodolite, level, electricity Magnetic wave rangefinder) from roadway leading forever a little and elevational point carry out connection survey acquisition.
In the present embodiment, above-mentioned steps 102 can use following preferred embodiment:
It is determined by following formula and is joined outside the rotation between laser radar sensor and Inertial Measurement Unit:
Wherein,For the amount of relative rotation of adjacent time inter Inertial Measurement Unit;For laser radar to be estimated Join outside rotation between sensor and Inertial Measurement Unit;For the relative rotation of adjacent time inter laser radar sensor Amount.
Preferably, region feature in laser radar point cloud is extracted, using region feature by the subgraph of present frame and the accumulation of building It is matched, data correlation is carried out using KNN (k-nearest neighbor algorithm) algorithm, by being fitted point of proximity Plane into subgraph is calculated using the distance of point to face as matching measurement
The method for judging whether rotation calibration process is completed is as follows:
Collect observation of the N to the Inertial Measurement Unit in the observation of laser radar sensor and corresponding time interval, construction Overdetermined equation:
Wherein:
Wherein,ForQuaternary number expression, QNFor overdetermination matrix, λiThe noise and outer dot factor of i-th pair observation are characterized,WithIt is respectivelyWithIt is first three element of quaternary number qxyzAntisymmetric matrix, qwIt is the of quaternary number Four elements, Ι3For 3 × 3 unit matrix;
Calculate overdetermination matrix QNThe second small singular value σmin2, with preset threshold value σthresholdIt is compared, such as Fruit meets σmin2> σthreshold, then overdetermined equation has solution, terminates rotation calibration process.
In the present embodiment, above-mentioned steps 103 can use following preferred embodiment:
First frame laser radar sensor observes the pose of corresponding Inertial Measurement Unit as starting point using in sliding window Local coordinate system is constructed, determines that used all the sensors are observed using sliding window, linear estimator is by minimizing institute There is the sum of the geneva norm of sensor observation residual error to determine variable to be estimated:
Wherein, χ is variable to be estimated:
Wherein, xn,xn+1,…,xn+NIndicate Inertial Measurement Unit state, includePositionWith the direction of gravity (roll angle and roll angle of IMU i.e. under world coordinate system), subscript n refer to Inertial Measurement Unit, and subscript N is current sliding window mouth The number of states of middle Inertial Measurement Unit;Join outside translation between laser radar sensor and Inertial Measurement Unit;For Join outside translation between UWB mobile node and Inertial Measurement Unit;pm,pm+1,…,pm+MIndicate UWB anchor node in local coordinate Position under system, subscript m refer to UWB observation, and subscript M is all available UWB observation quantity in current window;rPFor marginalisation Priori factor residual error item afterwards, HpFor the Jacobian matrix of priori factor residual error item after marginalisation;For inertia measurement Unit observes residual error item, is constructed using pre-integration method,For Inertial Measurement Unit observational equation,For corresponding association Variance matrix, B are that Inertial Measurement Unit pre-integration all in sliding window observes quantity;For laser radar sensing Device observes residual error item, by plane of the fitting point of proximity into subgraph, carries out structure as matching measurement using the distance of point to face It builds,For laser radar sensor observational equation, PsFor corresponding covariance matrix, A is that all and subgraph carries out matched thunder Take things philosophically the quantity of survey;Residual error item is observed for UWB, observes building using the distance of UWB mobile node to UWB anchor node,For UWB observational equation,For corresponding covariance matrix, C is that all UWB observe quantity in sliding window;
Solve linear equation:
PBLU) χ=(bP+bB+bL+bU)
Wherein, ΛB,bBIt is the information matrix and vector of Inertial Measurement Unit observation;ΛL,bLIt is that laser radar sensor is seen The information matrix and vector of survey;ΛU,bUIt is the information matrix and vector of UWB observation;Λp,bpIt is the information matrix of priori factor And vector;
Judge that whether convergent linear estimator method be as follows:
Calculate (ΛPBLU)-1Maximum singular value λmax, by itself and preset threshold λthresholdIt is compared, if λmax< λthresholdThen judge that linear estimator is restrained.
In the present embodiment, above-mentioned steps 104 can use following preferred embodiment:
The coordinate under UTM coordinate system is converted in the absolute coordinate under world coordinate system by each UWB anchor nodeSimultaneously Obtain the position estimation value P of each UWB anchor node under local coordinate systemi, construct error equation:
Wherein, M is that available UWB observes quantity in current sliding window,For the outer ginseng of local coordinate system and world coordinate system Transformation;
The M+1 position coordinates newly observed are added to above-mentioned error equation, obtain eM+1, utilize the condition of convergenceThe outer ginseng transformation of iterative solutionWherein, Δ is normalization variable, and ε is preset threshold.
The present invention have also been devised it is a kind of based under above-mentioned mine the method for calibrating external parameters of Multi-sensor Fusion it is accurate Localization method, as shown in Fig. 2, steps are as follows:
Step 201: using under mine Multi-sensor Fusion method for calibrating external parameters output outer ginseng estimated result as Initial value, building nonlinear optimization estimator carry out the Continuous optimization estimation of calibration result.
Step 202: utilizing pose and local coordinate system and generation of the underground mobile equipment of estimation under local coordinate system The outer ginseng transformation relation of boundary's coordinate system, determines pose of the underground mobile equipment under world coordinate system.
Step 203: using pose of the underground mobile equipment under world coordinate system, and the laser radar sensor obtained Point cloud information is registered under world coordinate system, obtains global map.
Step 204: being estimated using the pose of global map and the laser radar sensor under world coordinate system, contraposition Appearance information is integrated, and the accurate positioning result under global map is obtained;
Step 205: result will be accurately positioned and return to nonlinear optimization estimator, start to estimate next time.
In the present embodiment, above-mentioned steps 201 can use following preferred embodiment:
The state variable of the nonlinear optimization estimator is indicated using error state:
Wherein, δ indicates error symbol;
Nonlinear optimization estimator is estimated by minimizing the sum of the geneva norm of all observation residual errors to obtain maximum a posteriori Meter:
Wherein,For the Jacobian matrix of Inertial Measurement Unit observation, HsFor laser radar sensor observation Jacobian matrix,For the Jacobian matrix of UWB observation;
Solve nonlinear equation:
PBLU) δ χ=(bP+bB+bL+bU)
Above-mentioned nonlinear equation, error state renewal process are iteratively solved using nonlinear optimization estimator are as follows:WithFor the state variable that front and back iteratively solves twice,Operation is for the variable of theorem in Euclid space Simple add operation is the multiplying of quaternary number for rotation process.
In the present embodiment, above-mentioned steps 202 can use following preferred embodiment:
The pose for operating and determining underground mobile equipment under world coordinate system is integrated by the matrix of matrix multiplication
Wherein,For pose of the underground mobile equipment estimated by nonlinear estimator under local coordinate system,To estimate The local coordinate system of meter and the outer ginseng transformation relation of world coordinate system.
In the present embodiment, above-mentioned steps 203 can use following preferred embodiment:
During registering point cloud information to world coordinate system, using position of the underground mobile equipment under world coordinate system Appearance obtains the current point cloud information S of distortionless laser radar sensor as movement priorik, it is registered to world coordinate system Under existing global mapIn, obtain new global map
In the present embodiment, above-mentioned steps 204 can use following preferred embodiment:
Utilize global mapAnd it is transformed into the laser radar sensor point cloud information S under world coordinate systemk+1, utilize Sk+1It arrivesScan matching, calculate obtain build figure process pose transformationPose transformation to the acquisition of figure process is built With posture information of the underground mobile equipment under world coordinate systemIt is integrated, obtains the accurate positioning result under global map
Embodiment is merely illustrative of the invention's technical idea, and this does not limit the scope of protection of the present invention, it is all according to Technical idea proposed by the present invention, any changes made on the basis of the technical scheme are fallen within the scope of the present invention.

Claims (10)

1. the method for calibrating external parameters of Multi-sensor Fusion under a kind of mine, which is characterized in that the multisensor includes swashing Optical radar sensor, Inertial Measurement Unit and ultra wide band mould group, the ultra wide band mould group include UWB mobile node and several UWB Anchor node, the laser radar sensor, Inertial Measurement Unit and UWB mobile node are separately positioned on underground mobile equipment, Several UWB anchor nodes are distributed in different location in roadway;The method for calibrating external parameters is as follows:
(1) absolute position of each UWB anchor node under world coordinate system in roadway is obtained, the observation letter of each sensor is obtained Breath;
(2) control underground mobile equipment carries out determining laser radar sensor and inertia measurement list comprising the movement centainly rotated Join outside rotation between member, and judges automatically whether rotation calibration process is completed;
(3) linear estimator based on sliding window is constructed, the direction for estimating gravity, each UWB anchor node are under local coordinate system Position, join outside the translation between laser radar sensor and Inertial Measurement Unit, UWB mobile node and Inertial Measurement Unit it Between translation outside join, position and speed of the Inertial Measurement Unit under local coordinate system, and whether judge automatically linear estimator Convergence;
(4) position coordinates using the UWB anchor node estimated under local coordinate system, with UWB anchor node in world coordinate system Under absolute location coordinates be aligned, determine between local coordinate system and world coordinate system outer ginseng transformation;
(5) parameter value of all estimations under world coordinate system is exported.
2. according to claim 1 under mine Multi-sensor Fusion method for calibrating external parameters, which is characterized in that in step (1) in, the method for obtaining absolute position of each UWB anchor node under world coordinate system in roadway includes: to be believed using underground geography Breath system database is obtained, is obtained by field surveys and a little joined from leading in roadway forever with elevational point using measuring instrument System's measurement obtains.
3. according to claim 1 under mine Multi-sensor Fusion method for calibrating external parameters, which is characterized in that in step (2) it in, is determined by following formula and is joined outside the rotation between laser radar sensor and Inertial Measurement Unit:
Wherein,For the amount of relative rotation of adjacent time inter Inertial Measurement Unit;For laser radar sensor to be estimated Join outside rotation between Inertial Measurement Unit;For the amount of relative rotation of adjacent time inter laser radar sensor.
The method for judging whether rotation calibration process is completed is as follows:
Observation of the N to the Inertial Measurement Unit in the observation of laser radar sensor and corresponding time interval is collected, overdetermination is constructed Equation:
Wherein:
Wherein,ForQuaternary number expression, QNFor overdetermination matrix, λiThe noise and outer dot factor of i-th pair observation are characterized,WithIt is respectivelyWith It is first three element of quaternary number qxyzAntisymmetric matrix, qwFor the 4th member of quaternary number Element, Ι3For 3 × 3 unit matrix;
Calculate overdetermination matrix QNThe second small singular value σmin2, with preset threshold value σthresholdIt is compared, if full Sufficient σmin2> σthreshold, then overdetermined equation has solution, terminates rotation calibration process.
4. according to claim 1 under mine Multi-sensor Fusion method for calibrating external parameters, which is characterized in that in step (4) in, the coordinate under UTM coordinate system is converted in the absolute coordinate under world coordinate system by each UWB anchor nodeIt obtains simultaneously Take the position estimation value P of each UWB anchor node under local coordinate systemi, construct error equation:
Wherein, M is that available UWB observes quantity in current sliding window,For the transformation of the outer ginseng of local coordinate system and world coordinate system;
The M+1 position coordinates newly observed are added to above-mentioned error equation, obtain eM+1, utilize the condition of convergenceThe outer ginseng transformation of iterative solutionWherein, Δ is normalization variable, and ε is preset threshold.
5. according to claim 1 under mine Multi-sensor Fusion method for calibrating external parameters, which is characterized in that in step (3) in, first frame laser radar sensor observes the pose of corresponding Inertial Measurement Unit as starting point structure using in sliding window Local coordinate system is built, determines that used all the sensors are observed using sliding window, linear estimator is all by minimizing Sensor observes the sum of geneva norm of residual error and determines variable to be estimated:
Wherein, χ is variable to be estimated:
Wherein, xn,xn+1,…,xn+NIndicate Inertial Measurement Unit state, includePositionWith the direction of gravitySubscript N refers to Inertial Measurement Unit, and subscript N is the number of states of Inertial Measurement Unit in current sliding window mouth;For laser radar biography Join outside translation between sensor and Inertial Measurement Unit;Join outside translation between UWB mobile node and Inertial Measurement Unit; pm,pm+1,…,pm+MIndicate position of the UWB anchor node under local coordinate system, subscript m refers to UWB observation, and subscript M is to work as front window All available UWB observe quantity in mouthful;rPFor the priori factor residual error item after marginalisation, HpIt is residual for priori factor after marginalisation The Jacobian matrix of poor item;Residual error item is observed for Inertial Measurement Unit, is constructed using pre-integration method, For Inertial Measurement Unit observational equation,For corresponding covariance matrix, B is that Inertial Measurement Unit all in sliding window is pre- Integral observation quantity;Residual error item is observed for laser radar sensor, by being fitted plane of the point of proximity into subgraph, Distance using point to face is constructed as matching measurement,For laser radar sensor observational equation, PsFor corresponding association Variance matrix, A are all quantity that matched radar observation is carried out with subgraph;Residual error item is observed for UWB, utilizes UWB The distance of mobile node to UWB anchor node observes building,For UWB observational equation,For corresponding covariance matrix, C is to slide All UWB observe quantity in dynamic window;
Solve linear equation:
PBLU) χ=(bP+bB+bL+bU)
Wherein, ΛB,bBIt is the information matrix and vector of Inertial Measurement Unit observation;ΛL,bLIt is that laser radar sensor is observed Information matrix and vector;ΛU,bUIt is the information matrix and vector of UWB observation;Λp,bpBe priori factor information matrix and to Amount;
Judge that whether convergent linear estimator method be as follows:
Calculate (ΛPBLU)-1Maximum singular value λmax, by itself and preset threshold λthresholdIt is compared, if λmax < λthresholdThen judge that linear estimator is restrained.
6. according to claim 5 under mine the method for calibrating external parameters of Multi-sensor Fusion precise positioning method, It is characterized in that, comprising the following steps:
(a) using the outer ginseng estimated result of the method for calibrating external parameters output of Multi-sensor Fusion under mine as initial value, building Nonlinear optimization estimator carries out the Continuous optimization estimation of calibration result;
(b) pose and local coordinate system and world coordinate system using the underground mobile equipment of estimation under local coordinate system Outer ginseng transformation relation, determine pose of the underground mobile equipment under world coordinate system;
(c) pose of the underground mobile equipment under world coordinate system, and the laser radar sensor point cloud information obtained are utilized, It is registered under world coordinate system, obtains global map;
(d) estimated using the pose of global map and the laser radar sensor under world coordinate system, posture information is carried out It is integrated, obtain the accurate positioning result under global map;
(e) result will be accurately positioned and returns to nonlinear optimization estimator, start to estimate next time.
7. based on the precise positioning method of the method for calibrating external parameters of Multi-sensor Fusion under mine described in claim 6, It is characterized in that, in step (a), the state variable of the nonlinear optimization estimator is indicated using error state:
Wherein, δ indicates error symbol;
Nonlinear optimization estimator obtains MAP estimation by minimizing the sum of geneva norm of all observation residual errors:
Wherein,For the Jacobian matrix of Inertial Measurement Unit observation, HsFor the Jacobian of laser radar sensor observation Matrix,For the Jacobian matrix of UWB observation;
Solve nonlinear equation:
PBLU) δ χ=(bP+bB+bL+bU)
Above-mentioned nonlinear equation, error state renewal process are iteratively solved using nonlinear optimization estimator are as follows: WithFor the state variable that front and back iteratively solves twice,Operation is simple add operation for the variable of theorem in Euclid space, It is the multiplying of quaternary number for rotation process.
8. according to claim 6 under mine the method for calibrating external parameters of Multi-sensor Fusion precise positioning method, It is characterized in that, in step (b), is integrated by the matrix of matrix multiplication and operate determining underground mobile equipment under world coordinate system Pose
Wherein,For pose of the underground mobile equipment estimated by nonlinear estimator under local coordinate system,For estimation The outer ginseng transformation relation of local coordinate system and world coordinate system.
9. according to claim 6 under mine the method for calibrating external parameters of Multi-sensor Fusion precise positioning method, It is characterized in that, it is alive using underground mobile equipment during registering point cloud information to world coordinate system in step (c) Pose under boundary's coordinate system obtains the current point cloud information S of distortionless laser radar sensor as movement priorik, infused Volume is to global map existing under world coordinate systemIn, obtain new global map
10. according to claim 9 under mine the method for calibrating external parameters of Multi-sensor Fusion precise positioning method, It is characterized in that, in step (d), utilizes global mapAnd it is transformed into the laser radar sensor point under world coordinate system Cloud information Sk+1, utilize Sk+1It arrivesScan matching, calculate obtain build figure process pose transformationIt is obtained to figure process is built Pose transformationWith posture information of the underground mobile equipment under world coordinate systemIt is integrated, is obtained under global map Accurate positioning result
CN201910806408.0A 2019-08-29 2019-08-29 External parameter calibration and accurate positioning method for fusion of multiple sensors under mine Active CN110514225B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910806408.0A CN110514225B (en) 2019-08-29 2019-08-29 External parameter calibration and accurate positioning method for fusion of multiple sensors under mine

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910806408.0A CN110514225B (en) 2019-08-29 2019-08-29 External parameter calibration and accurate positioning method for fusion of multiple sensors under mine

Publications (2)

Publication Number Publication Date
CN110514225A true CN110514225A (en) 2019-11-29
CN110514225B CN110514225B (en) 2021-02-02

Family

ID=68628834

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910806408.0A Active CN110514225B (en) 2019-08-29 2019-08-29 External parameter calibration and accurate positioning method for fusion of multiple sensors under mine

Country Status (1)

Country Link
CN (1) CN110514225B (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111103578A (en) * 2020-01-10 2020-05-05 清华大学 Laser radar online calibration method based on deep convolutional neural network
CN111259807A (en) * 2020-01-17 2020-06-09 中国矿业大学 Underground limited area mobile equipment positioning system
CN111307174A (en) * 2020-02-05 2020-06-19 深圳前海达闼云端智能科技有限公司 Calibration method of sensor, moving object and storage medium
CN111443337A (en) * 2020-03-27 2020-07-24 北京航空航天大学 Radar-IMU calibration method based on hand-eye calibration
CN112362084A (en) * 2020-11-23 2021-02-12 北京三快在线科技有限公司 Data calibration method, device and system
CN112414431A (en) * 2020-11-18 2021-02-26 的卢技术有限公司 Robust vehicle-mounted multi-sensor external parameter calibration method
CN112540345A (en) * 2020-11-18 2021-03-23 广东工业大学 Dual-model positioning method and system for detecting UWB quality based on Gaussian distribution
CN112793629A (en) * 2021-03-04 2021-05-14 上海申传电气股份有限公司 Autonomous positioning method for underground coal mine explosion-proof storage battery rail electric locomotive
CN113470121A (en) * 2021-09-06 2021-10-01 深圳市普渡科技有限公司 Autonomous mobile platform, external parameter optimization method, device and storage medium
CN114199240A (en) * 2022-02-18 2022-03-18 武汉理工大学 Two-dimensional code, laser radar and IMU fusion positioning system and method without GPS signal
CN114322996A (en) * 2020-09-30 2022-04-12 阿里巴巴集团控股有限公司 Pose optimization method and device of multi-sensor fusion positioning system
CN114323003A (en) * 2021-12-27 2022-04-12 青岛慧拓智能机器有限公司 Underground mine fusion positioning method based on UMB, IMU and laser radar
CN114413887A (en) * 2021-12-24 2022-04-29 北京理工大学前沿技术研究院 Method, equipment and medium for calibrating external parameters of sensor
CN114485623A (en) * 2022-02-16 2022-05-13 东南大学 Camera-IMU-UWB fusion accurate positioning method for focusing distance
US11385177B2 (en) 2018-04-13 2022-07-12 Washington University Designs of accurate pm sensors and systems for laboratory and real time calibration / data inversion
CN116202511A (en) * 2023-05-06 2023-06-02 煤炭科学研究总院有限公司 Method and device for determining pose of mobile equipment under long roadway ultra-wideband one-dimensional constraint
CN116774195A (en) * 2023-08-22 2023-09-19 国网天津市电力公司滨海供电分公司 Excitation judgment and parameter self-adjustment method and system for multi-sensor combined calibration
RU2810800C1 (en) * 2021-03-04 2023-12-28 Шанхай Шэньчуань Электрик Ко., Лтд Autonomous positioning method for mine electric locomotive with explosion-proof battery
CN112881999B (en) * 2021-01-25 2024-02-02 上海西虹桥导航技术有限公司 Semi-automatic calibration method for multi-line laser radar and vision sensor
CN117528420A (en) * 2023-11-14 2024-02-06 滕州郭庄矿业有限责任公司 Mine personnel position detection system based on Internet of things

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106546262A (en) * 2016-10-11 2017-03-29 山东科技大学 Based on plane control and about binding close traverse measurement system external parameters calibration method
CN106908759A (en) * 2017-01-23 2017-06-30 南京航空航天大学 A kind of indoor pedestrian navigation method based on UWB technology
CN107180215A (en) * 2017-05-31 2017-09-19 同济大学 Figure and high-precision locating method are built in parking lot based on warehouse compartment and Quick Response Code automatically
CN107644229A (en) * 2017-08-31 2018-01-30 浙江工业大学 Indoor anomaly detection method based on multi-sensor information fusion
CN107850436A (en) * 2015-05-23 2018-03-27 深圳市大疆创新科技有限公司 Merged using the sensor of inertial sensor and imaging sensor
CN108225302A (en) * 2017-12-27 2018-06-29 中国矿业大学 A kind of petrochemical factory's crusing robot alignment system and method
CN109304730A (en) * 2017-07-28 2019-02-05 华中科技大学 A kind of robot kinematics' parameter calibration method based on laser range finder
CN109544638A (en) * 2018-10-29 2019-03-29 浙江工业大学 A kind of asynchronous online calibration method for Multi-sensor Fusion
CN109900265A (en) * 2019-03-15 2019-06-18 武汉大学 A kind of robot localization algorithm of camera/mems auxiliary Beidou

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107850436A (en) * 2015-05-23 2018-03-27 深圳市大疆创新科技有限公司 Merged using the sensor of inertial sensor and imaging sensor
CN106546262A (en) * 2016-10-11 2017-03-29 山东科技大学 Based on plane control and about binding close traverse measurement system external parameters calibration method
CN106908759A (en) * 2017-01-23 2017-06-30 南京航空航天大学 A kind of indoor pedestrian navigation method based on UWB technology
CN107180215A (en) * 2017-05-31 2017-09-19 同济大学 Figure and high-precision locating method are built in parking lot based on warehouse compartment and Quick Response Code automatically
CN109304730A (en) * 2017-07-28 2019-02-05 华中科技大学 A kind of robot kinematics' parameter calibration method based on laser range finder
CN107644229A (en) * 2017-08-31 2018-01-30 浙江工业大学 Indoor anomaly detection method based on multi-sensor information fusion
CN108225302A (en) * 2017-12-27 2018-06-29 中国矿业大学 A kind of petrochemical factory's crusing robot alignment system and method
CN109544638A (en) * 2018-10-29 2019-03-29 浙江工业大学 A kind of asynchronous online calibration method for Multi-sensor Fusion
CN109900265A (en) * 2019-03-15 2019-06-18 武汉大学 A kind of robot localization algorithm of camera/mems auxiliary Beidou

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
YUNBAO HUANG,ET AL: "Multi-Sensor Calibration through Iterative Registration and Fusion", 《COMPUTER AIDED DESIGN》 *
李雨潭,李猛钢,朱 华: "煤矿搜救机器人履带式行走机构性能评价体系", 《工程科学学报》 *
罗 斌等: "光学头部姿态跟踪的多传感器数据融合研究", 《自 动 化 学 报》 *

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11385177B2 (en) 2018-04-13 2022-07-12 Washington University Designs of accurate pm sensors and systems for laboratory and real time calibration / data inversion
CN111103578A (en) * 2020-01-10 2020-05-05 清华大学 Laser radar online calibration method based on deep convolutional neural network
CN111259807A (en) * 2020-01-17 2020-06-09 中国矿业大学 Underground limited area mobile equipment positioning system
CN111259807B (en) * 2020-01-17 2023-09-01 中国矿业大学 Underground limited area mobile equipment positioning system
CN111307174A (en) * 2020-02-05 2020-06-19 深圳前海达闼云端智能科技有限公司 Calibration method of sensor, moving object and storage medium
CN111443337B (en) * 2020-03-27 2022-03-08 北京航空航天大学 Radar-IMU calibration method based on hand-eye calibration
CN111443337A (en) * 2020-03-27 2020-07-24 北京航空航天大学 Radar-IMU calibration method based on hand-eye calibration
CN114322996A (en) * 2020-09-30 2022-04-12 阿里巴巴集团控股有限公司 Pose optimization method and device of multi-sensor fusion positioning system
CN114322996B (en) * 2020-09-30 2024-03-19 阿里巴巴集团控股有限公司 Pose optimization method and device of multi-sensor fusion positioning system
CN112540345A (en) * 2020-11-18 2021-03-23 广东工业大学 Dual-model positioning method and system for detecting UWB quality based on Gaussian distribution
CN112414431A (en) * 2020-11-18 2021-02-26 的卢技术有限公司 Robust vehicle-mounted multi-sensor external parameter calibration method
CN112540345B (en) * 2020-11-18 2023-08-01 广东工业大学 Dual-model positioning method and system for detecting UWB quality based on Gaussian distribution
CN112414431B (en) * 2020-11-18 2023-05-12 的卢技术有限公司 Robust vehicle-mounted multi-sensor external parameter calibration method
CN112362084A (en) * 2020-11-23 2021-02-12 北京三快在线科技有限公司 Data calibration method, device and system
CN112881999B (en) * 2021-01-25 2024-02-02 上海西虹桥导航技术有限公司 Semi-automatic calibration method for multi-line laser radar and vision sensor
CN112793629A (en) * 2021-03-04 2021-05-14 上海申传电气股份有限公司 Autonomous positioning method for underground coal mine explosion-proof storage battery rail electric locomotive
WO2022183870A1 (en) * 2021-03-04 2022-09-09 上海申传电气股份有限公司 Autonomous positioning method for underground coal mine anti-explosion storage battery rail electric locomotive
RU2810800C1 (en) * 2021-03-04 2023-12-28 Шанхай Шэньчуань Электрик Ко., Лтд Autonomous positioning method for mine electric locomotive with explosion-proof battery
CN113470121A (en) * 2021-09-06 2021-10-01 深圳市普渡科技有限公司 Autonomous mobile platform, external parameter optimization method, device and storage medium
CN113470121B (en) * 2021-09-06 2021-12-28 深圳市普渡科技有限公司 Autonomous mobile platform, external parameter optimization method, device and storage medium
CN114413887A (en) * 2021-12-24 2022-04-29 北京理工大学前沿技术研究院 Method, equipment and medium for calibrating external parameters of sensor
CN114413887B (en) * 2021-12-24 2024-04-02 北京理工大学前沿技术研究院 Sensor external parameter calibration method, device and medium
CN114323003A (en) * 2021-12-27 2022-04-12 青岛慧拓智能机器有限公司 Underground mine fusion positioning method based on UMB, IMU and laser radar
CN114485623B (en) * 2022-02-16 2024-02-23 东南大学 Focusing distance camera-IMU-UWB fusion accurate positioning method
CN114485623A (en) * 2022-02-16 2022-05-13 东南大学 Camera-IMU-UWB fusion accurate positioning method for focusing distance
CN114199240A (en) * 2022-02-18 2022-03-18 武汉理工大学 Two-dimensional code, laser radar and IMU fusion positioning system and method without GPS signal
CN116202511A (en) * 2023-05-06 2023-06-02 煤炭科学研究总院有限公司 Method and device for determining pose of mobile equipment under long roadway ultra-wideband one-dimensional constraint
CN116774195B (en) * 2023-08-22 2023-12-08 国网天津市电力公司滨海供电分公司 Excitation judgment and parameter self-adjustment method and system for multi-sensor combined calibration
CN116774195A (en) * 2023-08-22 2023-09-19 国网天津市电力公司滨海供电分公司 Excitation judgment and parameter self-adjustment method and system for multi-sensor combined calibration
CN117528420A (en) * 2023-11-14 2024-02-06 滕州郭庄矿业有限责任公司 Mine personnel position detection system based on Internet of things

Also Published As

Publication number Publication date
CN110514225B (en) 2021-02-02

Similar Documents

Publication Publication Date Title
CN110514225A (en) The calibrating external parameters and precise positioning method of Multi-sensor Fusion under a kind of mine
CN109945856B (en) Unmanned aerial vehicle autonomous positioning and mapping method based on inertia/radar
CN105547305B (en) A kind of pose calculation method based on wireless location and laser map match
Chen et al. A smart phone based PDR solution for indoor navigation
CN103472823B (en) A kind of grating map creating method of intelligent robot
CN102175887B (en) Mobile ultrasonic anemoclinograph and method for measuring wind speed and direction
CN103471595B (en) A kind of iteration expansion RTS mean filter method towards the navigation of INS/WSN indoor mobile robot tight integration
KR20190045915A (en) System and method for calibrating unstable sensors
CN112987065B (en) Multi-sensor-integrated handheld SLAM device and control method thereof
CN203687887U (en) Laser snow depth measurement instrument
CN107014375B (en) Indoor positioning system and method with ultra-low deployment
CN106680765A (en) INS/UWB pedestrian navigation system and method based on distributed combined filter
CN110026993B (en) Human body following robot based on UWB and pyroelectric infrared sensor
CN104808174B (en) Wireless positioning system of nuclear power station based on Kalman filter and dead reckoning
CN108168556B (en) Ultra-wideband positioning method for tunneling support bracket integrating particle swarm optimization and Taylor series expansion
CN102128625A (en) Initial matching method for use in gravimetric map matching in gravity-aided inertial navigation system
US20120120230A1 (en) Apparatus and Method for Small Scale Wind Mapping
CN103792515B (en) A kind of different platform 2 ties up radar and infrared sensor metric data synthetic method
CN111694001A (en) Real-time distance measurement positioning system for unmanned aerial vehicle
CN113763548A (en) Poor texture tunnel modeling method and system based on vision-laser radar coupling
CN105547282A (en) Method and measuring apparatus for mobile positioning target
CN113405560B (en) Unified modeling method for vehicle positioning and path planning
CN113324544B (en) Indoor mobile robot co-location method based on UWB/IMU (ultra wide band/inertial measurement unit) of graph optimization
CN107356902B (en) WiFi positioning fingerprint data automatic acquisition method
Jia et al. An indoor and outdoor seamless positioning system based on android platform

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant