CN117990112A - Unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering - Google Patents
Unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering Download PDFInfo
- Publication number
- CN117990112A CN117990112A CN202410403488.6A CN202410403488A CN117990112A CN 117990112 A CN117990112 A CN 117990112A CN 202410403488 A CN202410403488 A CN 202410403488A CN 117990112 A CN117990112 A CN 117990112A
- Authority
- CN
- China
- Prior art keywords
- matrix
- coordinate
- state
- time
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 100
- 238000001914 filtration Methods 0.000 title claims abstract description 27
- 239000011159 matrix material Substances 0.000 claims abstract description 101
- 230000008569 process Effects 0.000 claims abstract description 49
- 238000005070 sampling Methods 0.000 claims abstract description 44
- 238000004364 calculation method Methods 0.000 claims abstract description 25
- 239000013598 vector Substances 0.000 claims description 46
- 230000009466 transformation Effects 0.000 claims description 20
- 238000012545 processing Methods 0.000 claims description 14
- 230000007704 transition Effects 0.000 claims description 10
- 239000000203 mixture Substances 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 claims description 8
- 239000006185 dispersion Substances 0.000 claims description 6
- 230000026676 system process Effects 0.000 claims description 6
- 230000003044 adaptive effect Effects 0.000 claims description 5
- 230000001131 transforming effect Effects 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 6
- 230000002411 adverse Effects 0.000 abstract description 3
- 230000008859 change Effects 0.000 description 7
- 238000003384 imaging method Methods 0.000 description 7
- 238000002474 experimental method Methods 0.000 description 5
- 238000005259 measurement Methods 0.000 description 5
- 230000005693 optoelectronics Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 239000002245 particle Substances 0.000 description 2
- 206010034719 Personality change Diseases 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention relates to a target positioning method of an unmanned aerial vehicle photoelectric platform based on robust unscented Kalman filtering, aiming at a system disturbance scene subjected to complex environment interference, unscented Kalman Filtering (UKF) and a robust estimation theory are fused, and adverse effects of system interference on target positioning are reduced through real-time adjustment of dynamic residual errors. According to the method, the self-adaptive weight factors are dynamically adjusted through different residual errors at each moment, so that the weighted values of the prediction matrix and the error covariance are adjusted in real time, the influence of external environment changes on the stability of the system is reduced, and the positioning accuracy and the robustness of the system are improved. In addition, the algorithm reduces the calculation load by optimizing the sampling process, so that the real-time performance of the system is improved.
Description
Technical Field
The invention relates to the field of positioning filtering algorithms, in particular to an unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering.
Background
Unmanned aerial vehicle technology, which is excellent in economic cost effectiveness, maneuverability, timeliness and adaptability, has undergone rapid technological iterative development and has found wide application in a variety of fields. With the popularity of on-board optoelectronic systems, unmanned aerial vehicles have shown great potential in implementing target detection, tracking and positioning, and in particular in accurately acquiring target position information and motion characteristics, have attracted interest in many industries. Currently, unmanned aerial vehicles exhibit wide application potential, covering a plurality of fields such as site monitoring, meteorological detection, road inspection, geological exploration, flood monitoring, aerial photography, traffic control, electric power facility inspection, forest fire prevention and the like.
The airborne photoelectric platform forms a core component for unmanned aerial vehicle to execute tasks, and can accurately detect and position various targets on the ground and in the air through an integrated photoelectric system and photoelectric sensor technology. With the continuous progress of photoelectric technology and digital technology, the performance of the photoelectric system carried by the unmanned aerial vehicle is expected to be significantly improved under the promotion of miniaturization and high integration trend of the airborne system. This advance predicts a significant expansion of the detection range, an increase in sensitivity and resolution, and a further reduction in system mass and volume, which together provide solid hardware support for achieving high-precision localization of unmanned aerial vehicle targets.
Parameters including attitude (pitch, roll, heading angle), attitude of the imaging system, pitch angle, and focal length of the sensor are obtained by the on-board reconnaissance system, however, these data are not sufficient to directly determine the precise geographic location of the target. In order to resolve the three-dimensional position of the target in the geographic coordinate system, key information such as flight attitude data of the unmanned aerial vehicle, global Positioning System (GPS) coordinates, and visual axis orientation of the onboard photoelectric platform must be acquired at that time. Then, through a series of coordinate conversion processes, the three-dimensional space position of the target can be calculated. The working principle and the steps of the target positioning method based on the unmanned aerial vehicle carried with the photoelectric platform are shown in fig. 1.
The core of the passive target positioning method is the design of a target geographic position estimation algorithm. The complexity of target positioning is mainly due to the combined action of multiple factors of sensor precision, measuring tools, stability of a load bearing platform, navigation precision of an aircraft and positioning accuracy. In terms of accurate estimation methods of the target geographic location, international researchers are focusing on a variety of filtering techniques including kalman filtering (KALMAN FILTER, KF), extended kalman filtering (Extended KALMAN FILTER, EKF), volumetric kalman filtering (Cubature KALMAN FILTER, CKF), unscented kalman filtering (Unscented KALMAN FILTER, UKF), and particle filtering (PARTICLE FILTER, PF) algorithms. For example, the three-level EKF algorithm developed by Luo and colleagues significantly enhances positioning accuracy by establishing a geometric link between the target lateral position and its relative height. Xu et al generally predicts the starting position of the target using a digital altitude model, and subsequently refines the position estimate using CKF algorithms, especially in complex terrain, to improve the accuracy of positioning of the ground target. Furthermore, tang Da et al position the ground target through iteration UKF algorithm, have effectively overcome the slow and scattered difficult problem of filtering of tracking speed.
However, EKF, UKF and CKF algorithms require reasonable settings of the system noise matrix and the observed noise to suppress the noise of the system, are typically calibrated through extensive experimentation, and need to be set to a fixed value. Second, when the change in the operating conditions of the aircraft causes nonlinear disturbances in the photovoltaic system, time-varying noise that varies over time and does not have constant statistical properties is generated. Time-varying noise is largely divided into two types: one is gradual noise, whose statistical properties change slowly over a longer period of time; the other is abrupt noise, which is usually caused by abrupt changes in flight conditions, and the statistical characteristics change rapidly in a short time. The algorithms can not adaptively adjust parameters of the algorithms to process noise containing time-varying characteristics, and the algorithms are difficult to adapt to changing working environments in the flight process of the unmanned aerial vehicle, so that the target positioning accuracy of a photoelectric system is reduced.
Disclosure of Invention
The invention aims to solve the problems and requirements, and provides an unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering.
In order to solve the technical problems, the invention adopts the following technical scheme:
The unmanned aerial vehicle photoelectric platform target positioning method based on the robust unscented Kalman filtering is characterized by comprising the following steps of:
Step1, determining the initial position of a target point in a geodetic coordinate system, and determining the initial value of a state vector of the coordinate system And error covariance matrix initial/>；
Step 2, according to the position prior value of the target at the previous moment, carrying out real-time positioning updating on the target, and solving the state estimation value vector of the target at the moment t, and carrying out system state vector at the moment t-1Sampling the elements and the mean value and covariance thereof to generate n+1 sampling point sets/>；
Step 3, obtaining posterior estimation of the system state at the time t-1 through state transformation, wherein the specific formula is as follows:
；
In the method, in the process of the invention, For the system state vector updated from the time state at the ith sampling point time,/>Is the system process noise at time t-1; /(I)Is a state transition function;
weighting the mean value and variance of the state function to obtain a weighted mean value and a weighted covariance matrix of the state quantity of the system at the moment t ：
；
In the method, in the process of the invention,A system state quantity mean vector consisting of weighted means; /(I)Covariance matrix of process noise at time t,/>The weight of the ith sampling point;
Step 4, after transforming the system state quantity at the time t, transforming the observed value of the ith sampling point Performing weighted transformation to obtain a system observation mean value vector/>Wherein the calculation formula of each element is as follows:
；
Wherein, Is a nonlinear observation function;
Step5, utilizing the innovation vector Mean matrix of state quantity/>Sum covariance matrix/>Updating:
；
；
；
；
；
In the method, in the process of the invention, For the actual observation vector, the computer vision technology performs image tracking on the target to obtain; /(I)Is observation noise; /(I)A weighted covariance matrix for the observed quantity; /(I)A weighted covariance matrix for the state quantity and the observed quantity; /(I)A Kalman gain matrix for a moment; /(I)An estimated vector for the updated system state quantity; /(I)An estimated covariance matrix for the updated system state quantity;
Step 5, calculating a dynamic residual error:
；
；
；
；
；
In the method, in the process of the invention, For the system state matrix, the state quantity is used for estimating the mean value vector/>And observed quantity matrix/>Composition; /(I)For the system observation state transition matrix, the observation matrix/>And identity matrix/>Composition,/>Is a nonlinear observation function/>Jacobian matrix of (a); /(I)Is a system noise matrix, which is formed by a process noise matrix/>And an estimated covariance matrix of system state quantity/>Composition; /(I)Is an intermediate variable; /(I)A dynamic residual error at the time t;
Step 6, robustness processing:
Calculating an adaptive scaling factor
In the method, in the process of the invention,For/>Is an equivalent weight matrix of (1); i and j are the upper and lower calculation limits of the scale factors;
carrying out robustness processing on the estimated value of the state quantity and the estimated covariance by using an robust estimation theory to obtain a final state estimated value vector at the moment t:
；
；
In the method, in the process of the invention, The final state estimation value vector subjected to robustness processing at the moment t can be used for iterative estimation of the system state of the target at the next moment; /(I)Covariance matrix of final system estimation value at t moment is subjected to robustness processing/>Will be at/>And correcting the state estimation value at the moment.
Further, in the step 1,
In the method, in the process of the invention,、/>、/>The initial variances of the photoelectric platform system in the L coordinate, the B coordinate and the H coordinate are respectively/>、And/>The initial longitude, latitude and geodetic elevation of the target, respectively.
Further, the specific sampling process in step 2 is as follows:
In the method, in the process of the invention, Initial weights for sampling points; /(I)To scale the parameters according to/>The calculation is used for controlling the dispersion degree of the sampling points; n is the number of sampling points; /(I)For a matrix of n rows and n columns, each element is 1; /(I)Is an n-dimensional identity matrix; /(I)Is by/>And/>A matrix obtained by calculation is used for adjusting the dispersion of sampling points; /(I)Is a diagonal element,/>Is a diagonal matrix of weights, used for updating sampling points; /(I)For/>A state covariance matrix of the moment; /(I)For a column vector of row 1 and column, each element is/>；/>For/>The ith element of the time sampling point set; /(I)Is the mean vector when there is only one sampling point, obviously; /(I),/>The calculation formula of the mean vector for the ith sampling point is as follows:
。
Further, in step 3, the state transition function The formula of (2) is:
；
Wherein: A state transition matrix from the time t-1 to the time t; /(I) A noise driving matrix from the time t-1 to the time t; /(I)Is the system process noise at time t-1.
Further, h (·) is a nonlinear function observed by the optoelectronic platform, and the specific formula of h (x) is:
x is the coordinate of the target point in the GCF coordinate system, let x= [ L, B, H ] T, L be longitude information, B be latitude information, H be elevation information, then the coordinate of the target point in the C coordinate system is [ x c ,yc ,zc]T, then:
The calculation formula of (x c ,yc ,zc) is:
；
wherein u 'refers to the coordinate of the object measured by the image tracker in the x direction of the pixel coordinate system, v' refers to the coordinate of the object measured by the image tracker in the y direction of the pixel coordinate system,
C x is the coordinate of the pixel plane center in the x direction, c y is the coordinate of the pixel plane center in the y direction; equivalent focal length、/>Camera focal length/>, respectivelyLateral and longitudinal dimensions of the pixels/>、/>Ratio of; /(I)For the coordinate transformation matrix from the HRD coordinate system to the C coordinate system,/>The coordinate transformation matrix from the NED coordinate system to the HRD coordinate system; /(I)Is the coordinate transformation matrix from ECEF coordinate system to NED coordinate systemA first eccentricity that is an ellipsoidal meridian ellipse of the earth; radius of curvature of the circle of mortise and tenon, wherein/> Is an ellipsoidal long half axle,/>Is an ellipsoidal short half shaft, and the units are meters.
After the technical scheme is adopted, compared with the prior art, the invention has the following advantages:
Aiming at a system disturbance scene subjected to complex environment interference, unscented Kalman Filtering (UKF) and a robust estimation theory are fused, and adverse effects of system interference on target positioning are reduced through real-time adjustment of dynamic residual errors. According to the method, the self-adaptive weight factors are dynamically adjusted through different residual errors at each moment, so that the weighted values of the prediction matrix and the error covariance are adjusted in real time, the influence of external environment changes on the stability of the system is reduced, and the positioning accuracy and the robustness of the system are improved. In addition, the invention reduces the calculation load by optimizing the sampling process, thereby improving the real-time performance of the system.
Drawings
FIG. 1 is a target positioning workflow based on an unmanned airborne optoelectronic platform;
FIG. 2 is a schematic diagram of GCF, ECEF and NED coordinate systems;
FIG. 3 is an imaging model of an optoelectronic platform;
Fig. 4 is a schematic diagram of experimental unmanned aerial vehicle flight trajectory planning;
FIG. 5 is an imaged image of a different waypoint drone;
FIG. 6 is a graph of the target positioning results for tracks W1 through W2;
Fig. 7 is a target positioning result diagram of the trajectories W3 to W4.
Detailed Description
The principles and features of the present invention are described below with reference to the drawings, the examples are illustrated for the purpose of illustrating the invention and are not to be construed as limiting the scope of the invention.
In order to meet the requirement of an unmanned aerial vehicle photoelectric system on real-time target positioning, the research provides a self-adaptive robust unscented Kalman filtering method (Adaptive Robust Unscented KALMAN FILTER, ARUKF) based on dynamic residual feedback. Aiming at a system disturbance scene subjected to complex environment interference, the method fuses Unscented Kalman Filtering (UKF) and a robust estimation theory, and reduces adverse effects of system interference on target positioning through real-time adjustment of dynamic residual errors. According to the method, the self-adaptive weight factors are dynamically adjusted through different residual errors at each moment, so that the weighted values of the prediction matrix and the error covariance are adjusted in real time, the influence of external environment changes on the stability of the system is reduced, and the positioning accuracy and the robustness of the system are improved. In addition, the algorithm reduces the calculation load by optimizing the sampling process, so that the real-time performance of the system is improved.
The object positioning process mainly involves 6 basic inertial coordinate systems:
(1) Geodetic coordinate system GCF
(2) Geocentric coordinate system ECEF
(3) Navigation coordinate system NED
(4) Organism coordinate system HRD
(5) Camera coordinate system C
(6) Image physical coordinate system O
(7) Image coordinate system I
The GCF coordinate system, the ECEF coordinate system and the NED coordinate system are shown in FIG. 2.
The imaging model of the optoelectronic platform is shown in fig. 3, and mainly relates to HRD, C, O and I coordinate systems.
Assuming that a photovoltaic system mounted on an unmanned aerial vehicle performs continuous multiple observation activities on a plurality of targets in a field of view, since the system mainly estimates geographic position information of the targets, a system state quantity is set as geographic coordinates of the targets.
（1）
Wherein:、/> And/> Longitude, latitude and geodetic elevation of the time of day target, respectively. The state equation of the system can be expressed as
（2）
Wherein: subscripts k and k-1 represent times k and k-1, respectively; a state transition matrix from k-1 time to k time; /(I) Is a noise driving matrix; /(I)Is the system process noise covariance matrix. For stationary targets it is apparent/>And/>Take the value of
（3）
Wherein: For system noise/> Is a covariance matrix of (a).
In the photoelectric platform imaging model shown in FIG. 3, the body coordinate system HRD isDefining the optical center of a camera as the imaging center of the unmanned aerial vehicle, and establishing a camera coordinate system C/>, of a light spot platform。
The coordinates of the object in the C coordinate system areThe imaging model of the photoelectric platform can be known as follows:
（4）
Wherein u 'refers to the coordinate of the target measured by the image tracker in the x direction of the pixel coordinate system, v' refers to the coordinate of the target measured by the image tracker in the y direction of the pixel coordinate system, c x refers to the coordinate of the center of the pixel plane in the x direction, and c y refers to the coordinate of the center of the pixel plane in the y direction; equivalent focal length 、/>Camera focal length/>, respectivelyLateral and longitudinal dimensions of the pixels/>、/>Ratio of;
Assuming that the ground height of the photographing target area is the following equation:
（5）；
In the method, in the process of the invention, For the coordinate transformation matrix from the HRD coordinate system to the C coordinate system,/>The coordinate transformation matrix from the NED coordinate system to the HRD coordinate system; /(I)Is the coordinate transformation matrix from ECEF coordinate system to NED coordinate systemA first eccentricity that is an ellipsoidal meridian ellipse of the earth; /(I)Radius of curvature of circle of mortise and tenon, whereinIs an ellipsoidal long half axle,/>Is an ellipsoidal short half shaft, and the units are meters.
In order to meet the requirement of high-precision positioning of the target, the camera is used as an external measuring sensor. The observed quantity is the pixel coordinate of the target in the acquired image, and the mathematical expression is:
（6）
Then the observation equation is （7）；
Wherein: For measuring noise; /(I) Is a nonlinear function of the observation of the photoelectric platform. In addition,/>Is/>。
The unmanned aerial vehicle can encounter severe nonlinear attitude change during autonomous flight, and by adopting an Unscented Kalman Filter (UKF) algorithm, the problem of nonlinear observation common in the process of positioning an object of a photoelectric system can be effectively solved by means of an unscented transformation technology, and the unmanned aerial vehicle has excellent processing capacity on complex dynamic behaviors. The application of the UKF algorithm requires presetting of system and observation noise parameters, however, due to uncertainty of the unmanned aerial vehicle operating environment and fluctuation of wind speed, in addition to continuous vibration of the machine body in flight, accurate prediction of the distribution or type of noise errors becomes extremely difficult, which results in failure to construct an accurate function or random model. Therefore, it is important to reduce the influence of parameter update during the filtering process.
In view of the above, the present invention determines an adaptive weight factor by calculating a dynamic residual between a system observation and a predicted value. These weighting factors are used to weight the prediction state vector and its prediction error covariance at each moment to achieve an optimal balance between the dynamic model and the measured data. The method allows real-time adjustment of the observation noise matrix to reduce the influence of the disagreement between the preset noise parameters and the actual conditions, and effectively reduces the risk of divergence of the predicted position information in the filtering process, thereby improving the accuracy and the robustness of the algorithm. In addition, by employing a minimization strategy sampling, the present study further reduces the computational requirements of the system, increasing its real-time processing capacity. The specific algorithm implementation mainly comprises 6 steps:
(1) Setting a system state vector initial value and an error covariance matrix initial value, and providing a deterministic system response basis for subsequent system state space iteration, wherein the specific formula is as follows
（8）；
In the method, in the process of the invention,、/>、/>The initial variances of the photoelectric platform system in the L coordinate, the B coordinate and the H coordinate are respectively.
(2) The system state quantity is sampled and weighted. The photoelectric platform has extremely high requirements on algorithm instantaneity when observing the ground target, so that the UKF can greatly reduce the calculated amount by means of a minimum sampling strategy when generating sigma sampling points, and the operation efficiency and the positioning instantaneity are improved. First, according to the system state vectorSum covariance/>Generate/>Set of sample points/>. The specific sampling process is as follows:
（9）；
In the method, in the process of the invention, Initial weights for sampling points; /(I)To scale the parameters according to/>The calculation is used for controlling the dispersion degree of the Sigma sampling points; n is the number of sampling points;
（10）
In the method, in the process of the invention, For a matrix of n rows and n columns, each element is 1; /(I)Is an n-dimensional identity matrix; /(I)Is by/>And/>A matrix obtained by calculation is used for adjusting the dispersion of Sigma points;
（11）；
In the method, in the process of the invention, Is a diagonal element,/>Is a diagonal matrix of weights used for updating Sigma points; the calculation formula of the ith diagonal element is
（12）;
（13）；
In the method, in the process of the invention,For/>A state covariance matrix of the moment; /(I)For a column vector of row 1 and column, each element is/>；/>For/>The ith element of the time sampling point set; /(I)Is the mean vector when there is only one sampling point, obviously; /(I),/>The calculation formula of the mean vector for the ith sampling point is as follows:
（14）；
equations (9) through (14) describe the detailed steps of how Sigma samples are generated from the state mean vector and covariance matrix, primarily to capture the nonlinear characteristics of the predicted state distribution, and then obtain a posterior estimate of the system state through state transformation. The specific formula is
（15）；
In the method, in the process of the invention,Is the first element of the time system state quantity,/>Is the system process noise at time t-1; /(I)Is a state transition function;
weighting the mean value and variance of the state function to obtain a weighted mean value and a weighted covariance matrix of the state quantity of the system at the moment t ：
（16）
（17）；
In the method, in the process of the invention,A system state vector updated from the time state at the ith Sigma point moment; /(I)A system state quantity mean vector consisting of weighted means; /(I)A covariance matrix of process noise at the time t;
weighting the mean value and the variance of the state function to obtain a weighted mean value of the state quantity of the system And weighted covariance matrix/>：
(3) Observational quantity weighted transformation
After transforming the system state quantity at the time t, the ith observation value is transformedPerforming weighted transformation to obtain a system observation mean value vector/>Wherein the calculation formula of each element is as follows:
（18）；
(4) System state quantity update
Using innovation vectorsMean matrix of state quantity/>Sum covariance matrix/>Updating:
（19）；
（20）；
（21）；
（22）；
（23）；
In the method, in the process of the invention, The method comprises the steps of performing image tracking on a target by a computer vision technology to obtain an actual observation vector; /(I)Is observation noise; /(I)A weighted covariance matrix for the observed quantity; /(I)A weighted covariance matrix for the state quantity and the observed quantity; /(I)A Kalman gain matrix for a moment; /(I)An estimated vector for the updated system state quantity; /(I)An estimated covariance matrix for the updated system state quantity;
(5) Calculating a dynamic residual:
（24）；
（25）；
（26）；
（27）；
（28） ；
In the method, in the process of the invention, For the system state matrix, the state quantity is used for estimating the mean value vector/>And observed quantity matrix/>Composition; /(I)For the system observation state transition matrix, the observation matrix/>And identity matrix/>Composition,/>Is a nonlinear observation function/>Jacobian matrix of (a); /(I)Is a system noise matrix, which is formed by a process noise matrix/>And an estimated covariance matrix of system state quantity/>Composition; Is an intermediate variable; /(I) A dynamic residual error at the time t;
(6) Robust processing
Calculating an adaptive scaling factor
（29）
In the method, in the process of the invention,For/>Is an equivalent weight matrix of (1); i and j are the upper and lower calculation limits of the scale factors;
In order to reduce the influence of inaccurate preset noise on a system, robust processing is carried out on the estimated value and the estimated covariance of the state quantity by utilizing the robust estimation theory, and a final state estimated value vector at the moment t is obtained:
（30）；
（31）；
In the method, in the process of the invention, The final state estimated value vector subjected to robustness processing at the moment t; /(I)Covariance matrix of final system estimation value at t moment is subjected to robustness processing/>The state estimate is corrected at time t + 1. Secondly, the finally output system state estimation value can be obtained through iteration.
Photoelectric system target positioning experiment
The flight heights of the experimental unmanned aerial vehicle are 750m and 1500m respectively, and a flight path planning schematic diagram is shown in fig. 4. The unmanned plane starts to fly stably and linearly at the navigation point W 1, reaches W 4 after 3 fast turns, and completes continuous imaging observation for the target area for multiple times, wherein the observation frequency is 30 frames/second. 2 passing points, W 2 and W 3 respectively, are arranged in the flight track, and when the unmanned aerial vehicle makes a quick turn on the red track, the flight working condition can be changed obviously. The sampling frequency of the unmanned aerial vehicle photoelectric system is 5 frames/second, the total time from the flight point W 1 to the flight point W 4 is 35 seconds, and 7-9 seconds are the time range of the unmanned aerial vehicle for making a small-range turn. The imaging images of the target area by the unmanned aerial vehicle at different waypoints in the experiment are shown in fig. 5. As can be seen from fig. 5, 7 targets are arranged on the ground in the experiment, and the specific positional relationship is shown in the last image of fig. 5. In the observation process of the photoelectric system, the camera always ensures that all target points are in an observation range. Thus, from T1 to T7, the distance of the target point with respect to the imaging center gradually increases. The GPS coordinates of the 3 targets were precisely measured using a differential positioning system, and the results are shown in table 1. The standard deviation of the measurement error of the system is smaller than 0.1m, so that the measurement result can be used as true values of 3 target geographic positions.
Table 1 true geographic location of objects
In order to study the relation between the unmanned aerial vehicle flight condition and the proposed algorithm target positioning accuracy, observation data of flight trajectories W 1 to W 2 and flight trajectories W 3 to W 4 are selected for result comparison analysis, and the flight height is 750m. Taking the target T 1 as an example, the positioning results obtained using the trajectories W 1 to W 2 are shown in fig. 6, and the positioning results obtained using the trajectories W 3 to W 4 are shown in fig. 7. In fig. 6 and fig. 7, red five-pointed star is a true value of a target geographic position, yellow dots are filter initial values, black curves are geographic position measurement values at different times, gradient color curves are geographic position estimation values at different times, and the change of color from blue to red represents an increase of sampling time.
As can be seen from fig. 6 and 7, the ARUKF algorithm obtains a distance filtering value that gradually approaches to a true value over time. Wherein, the target distance estimation error of the flight tracks W 1 to W 2 in the north direction is 7.37m under the machine body coordinate system, and the target distance estimation error in the east direction is 7.92m; the target distance estimation error of the flight trajectories W 3 to W 4 in the north direction under the body coordinate system is 5.52m, and the target distance estimation error in the east direction is 3.72m.
For flight trajectories W 1 to W 2, the drone makes a fast maneuver turn at 7-9 seconds, the east-directed measurement is dithered to a small extent, and then the distance filtered value is slightly off the true value. The reason is that the unmanned aerial vehicle working condition is obviously changed due to maneuvering turning, nonlinear disturbance of a photoelectric system is aggravated, and estimation error of a target position is increased. However, ARUKF algorithm can still be converged to a true value rapidly in 10s, which shows that the algorithm can track the change of the observation noise rapidly and accurately, and effectively overcomes the mutation of the observation noise, thereby improving the positioning accuracy of the photoelectric platform. As can be seen from fig. 6 and 7, the estimated value of the position location obtained by ARUKF algorithm is also continuously approaching to the true value, which verifies the feasibility of the algorithm. For flight trajectories W 3 to W 4, the unmanned aerial vehicle is in stable flight in the whole course, the ARUKF algorithm can quickly converge to a true value in 10s, and the positioning accuracy is higher.
In order to study the performance advantages of the proposed algorithm for multi-target passive positioning, under the experimental situations that the conditions are set to be 750m in flight height, W 1 to W 2 in flight track and 20 times of experiments, the EKF algorithm, UKF algorithm, AEKF algorithm, PF algorithm and the proposed ARUKF algorithm are respectively utilized to calculate the average positioning error, the positioning root mean square error and the average operation time in each sampling period of all targets, and the results are shown in Table 2.
Table 2 positioning accuracy and calculation time of different algorithms
From table 2 the following conclusions can be drawn: (1) The EKF can not adaptively adjust the noise matrix, so that the average positioning error is larger and is between 46.32m and 63.10m, and the lowest positioning precision is shown; UKF processes the nonlinear problem by adopting lossless transformation, so that the positioning accuracy is improved to a certain extent, but the influence of time-varying noise cannot be effectively overcome; the AEKF can adaptively adjust the noise matrix, so that the positioning accuracy is further improved; the average positioning errors of PF and ARUKF on all targets differ by only 0.59m, and the overall difference of the positioning accuracy of PF and ARUKF is not large; (2) ARUKF has a calculation time of 0.039ms, which is similar to AEKF, is slightly higher than EKF, but is far lower than PF and UKF, the calculation efficiency is improved by 51% compared with PF, and the result is consistent with analysis in introduction and theoretical calculation complexity [32]; (3) Under the condition that the unmanned aerial vehicle turns, ARUKF shows stronger robustness on positioning root mean square error, the error range is between 2.24 meters and 6.21 meters, and the positioning effect is equivalent to that of the PF; (4) The positioning errors of the targets T5, T6 and T7 are generally larger than T1, T2, T3 and T4, because the targets T5, T6 and T7 are further from the imaging center, and the distortion of the nacelle lens results in a larger positioning error.
Aiming at the time-varying characteristic of the observation noise of the unmanned aerial vehicle photoelectric system, the invention provides a dynamic estimation method of the self-adaptive robust Kalman filtering algorithm for the target geographic position of the photoelectric platform. The algorithm is verified and tested in multiple aspects through the actually measured flight experiment, and the experimental result shows that:
(1) The maneuvering turning causes the unmanned aerial vehicle to change obviously, aggravates the nonlinear disturbance of the photoelectric system, and increases the estimation error of the target position. However, ARUKF algorithm can still be converged to a true value rapidly in 10s, which shows that the algorithm can track the change of time-varying observation noise rapidly and accurately, and effectively overcomes the variability of the observation noise;
(2) Under the condition that the flight trajectories are the same, the distance measurement value shows more remarkable fluctuation along with the increase of the flight altitude, and meanwhile, the error of target positioning by the algorithm is increased;
(3) According to the comparison analysis of the precision, the robustness and the efficiency of 5 different algorithms, the provided algorithm can reach the effect of PF in the aspects of precision and stability, and the efficiency is improved by 51% compared with PF.
The foregoing is illustrative of the best mode of carrying out the invention, and is not presented in any detail as is known to those of ordinary skill in the art. The protection scope of the invention is defined by the claims, and any equivalent transformation based on the technical teaching of the invention is also within the protection scope of the invention.
Claims (5)
1. The unmanned aerial vehicle photoelectric platform target positioning method based on the robust unscented Kalman filtering is characterized by comprising the following steps of:
Step1, determining the initial position of a target point in a geodetic coordinate system, and determining the initial value of a state vector of the coordinate system And error covariance matrix initial/>；
Step 2, according to the position prior value of the target at the previous moment, carrying out real-time positioning updating on the target, and solving the state estimation value vector of the target at the moment t, and carrying out system state vector at the moment t-1Sampling the elements and the mean value and covariance thereof to generate n+1 sampling point sets/>；
Step 3, obtaining posterior estimation of the system state at the time t-1 through state transformation, wherein the specific formula is as follows:
；
In the method, in the process of the invention, For the system state vector updated from the time state at the ith sampling point time,/>Is the system process noise at time t-1; /(I)Is a state transition function;
weighting the mean value and variance of the state function to obtain a weighted mean value and a weighted covariance matrix of the state quantity of the system at the moment t ：
；
In the method, in the process of the invention,A system state quantity mean vector consisting of weighted means; /(I)Covariance matrix of process noise at time t,/>The weight of the ith sampling point;
Step 4, after transforming the system state quantity at the time t, transforming the observed value of the ith sampling point Performing weighted transformation to obtain a system observation mean value vector/>Wherein the calculation formula of each element is as follows:
；
Wherein, Is a nonlinear observation function;
Step5, utilizing the innovation vector Mean matrix of state quantity/>Sum covariance matrix/>Updating:
；
；
；
；
；
In the method, in the process of the invention, For the actual observation vector, the computer vision technology performs image tracking on the target to obtain; /(I)Is observation noise; A weighted covariance matrix for the observed quantity; /(I) A weighted covariance matrix for the state quantity and the observed quantity; /(I)A Kalman gain matrix for a moment; /(I)An estimated vector for the updated system state quantity; /(I)An estimated covariance matrix for the updated system state quantity;
Step 5, calculating a dynamic residual error:
；
；
；
；
；
In the method, in the process of the invention, For the system state matrix, the state quantity is used for estimating the mean value vector/>And observed quantity matrix/>Composition; /(I)For the system observation state transition matrix, the observation matrix/>And identity matrix/>Composition,/>Is a nonlinear observation function/>Jacobian matrix of (a); /(I)Is a system noise matrix, which is formed by a process noise matrix/>And an estimated covariance matrix of system state quantity/>Composition; /(I)Is an intermediate variable; /(I)A dynamic residual error at the time t;
Step 6, robustness processing:
Calculating an adaptive scaling factor
In the method, in the process of the invention,For/>Is an equivalent weight matrix of (1); i and j are the upper and lower calculation limits of the scale factors;
carrying out robustness processing on the estimated value of the state quantity and the estimated covariance by using an robust estimation theory to obtain a final state estimated value vector at the moment t:
；
；
In the method, in the process of the invention, The final state estimation value vector subjected to robustness processing at the moment t can be used for iterative estimation of the system state of the target at the next moment; /(I)Covariance matrix of final system estimation value at t moment is subjected to robustness processing/>Will be at/>And correcting the state estimation value at the moment.
2. The unmanned aerial vehicle photoelectric platform target positioning method based on the robust unscented Kalman filtering according to claim 1, wherein in step 1,
In the method, in the process of the invention,、/>、/>The initial variances of the photoelectric platform system in the L coordinate, the B coordinate and the H coordinate are respectively/>、/>AndThe initial longitude, latitude and geodetic elevation of the target, respectively.
3. The unmanned aerial vehicle photoelectric platform target positioning method based on the robust unscented Kalman filtering according to claim 1, wherein the specific sampling process in the step 2 is as follows:
In the method, in the process of the invention, Initial weights for sampling points; /(I)To scale the parameters according to/>The calculation is used for controlling the dispersion degree of the sampling points; n is the number of sampling points; /(I)For a matrix of n rows and n columns, each element is 1; /(I)Is an n-dimensional identity matrix; /(I)Is by/>And/>A matrix obtained by calculation is used for adjusting the dispersion of sampling points; /(I)Is a diagonal element,/>Is a diagonal matrix of weights, used for updating sampling points; /(I)For/>A state covariance matrix of the moment; /(I)For a column vector of row 1 and column, each element is/>；/>For/>The ith element of the time sampling point set; /(I)Is the mean vector when there is only one sampling point, obviously; /(I),/>The calculation formula of the mean vector for the ith sampling point is as follows:
。
4. The unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering according to claim 1, wherein the state transformation function in step 3 is The formula of (2) is:
；
Wherein: A state transition matrix from the time t-1 to the time t; /(I) A noise driving matrix from the time t-1 to the time t; /(I)Is the system process noise at time t-1.
5. The unmanned aerial vehicle photoelectric platform target positioning method based on the robust unscented Kalman filtering according to claim 1, wherein h (·) is a nonlinear function observed by the photoelectric platform, and the specific formula of h (x) is:
x is the coordinate of the target point in the GCF coordinate system, let x= [ L, B, H ] T, L be longitude information, B be latitude information, H be elevation information, then the coordinate of the target point in the C coordinate system is [ x c ,yc,zc]T, then:
The calculation formula of (x c ,yc ,zc) is:
；
wherein u 'refers to the coordinate of the object measured by the image tracker in the x direction of the pixel coordinate system, v' refers to the coordinate of the object measured by the image tracker in the y direction of the pixel coordinate system,
C x is the coordinate of the pixel plane center in the x direction, c y is the coordinate of the pixel plane center in the y direction; equivalent focal length、/>Camera focal length/>, respectivelyLateral and longitudinal dimensions of the pixels/>、/>Ratio of; /(I)For the coordinate transformation matrix from the HRD coordinate system to the C coordinate system,/>The coordinate transformation matrix from the NED coordinate system to the HRD coordinate system; /(I)Is the coordinate transformation matrix from ECEF coordinate system to NED coordinate systemA first eccentricity that is an ellipsoidal meridian ellipse of the earth; /(I)Radius of curvature of the circle of mortise and tenon, wherein/>Is an ellipsoidal long half axle,/>Is an ellipsoidal short half shaft, and the units are meters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410403488.6A CN117990112B (en) | 2024-04-03 | 2024-04-03 | Unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410403488.6A CN117990112B (en) | 2024-04-03 | 2024-04-03 | Unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117990112A true CN117990112A (en) | 2024-05-07 |
| CN117990112B CN117990112B (en) | 2024-07-02 |
Family
ID=90896388
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410403488.6A Active CN117990112B (en) | 2024-04-03 | 2024-04-03 | Unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117990112B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120022784A1 (en) * | 2010-06-25 | 2012-01-26 | Thales | Navigation filter for a navigation system using terrain correlation |
| JP2013061309A (en) * | 2011-09-15 | 2013-04-04 | Yamaha Corp | Kalman filter, state estimation device, method for controlling kalman filter, and control program of kalman filter |
| JP2015219714A (en) * | 2014-05-16 | 2015-12-07 | 日本電信電話株式会社 | State estimation device, method, and program |
| CN109990786A (en) * | 2019-02-28 | 2019-07-09 | 深圳大学 | Maneuvering target tracking method and device |
| CN110196443A (en) * | 2019-06-06 | 2019-09-03 | 中国人民解放军战略支援部队信息工程大学 | A kind of fault-tolerance combined navigation method and system of aircraft |
| CN110728716A (en) * | 2019-09-04 | 2020-01-24 | 深圳市道通智能航空技术有限公司 | Calibration method and device and aircraft |
| US20200132775A1 (en) * | 2017-06-14 | 2020-04-30 | Mitsubishi Electric Corporation | State estimation device |
| CN111896008A (en) * | 2020-08-20 | 2020-11-06 | 哈尔滨工程大学 | Improved robust unscented Kalman filtering integrated navigation method |
| CN112683274A (en) * | 2020-12-22 | 2021-04-20 | 西安因诺航空科技有限公司 | Unmanned aerial vehicle integrated navigation method and system based on unscented Kalman filtering |
| US20220065587A1 (en) * | 2020-08-31 | 2022-03-03 | Raytheon Company | System and method of hypersonic object tracking |
| WO2022126397A1 (en) * | 2020-12-15 | 2022-06-23 | 深圳市大疆创新科技有限公司 | Data fusion method and device for sensor, and storage medium |
| WO2022205526A1 (en) * | 2021-04-01 | 2022-10-06 | 江苏科技大学 | Dynamic positioning information fusion method for unmanned underwater vehicle cluster |
| CN115981148A (en) * | 2022-12-07 | 2023-04-18 | 哈尔滨工业大学 | Unmanned aerial vehicle ground moving target tracking method |
| CN116520380A (en) * | 2023-04-28 | 2023-08-01 | 南京邮电大学 | Dynamic target combined positioning method based on self-adaptive Kalman filtering |
| US20230297642A1 (en) * | 2022-03-18 | 2023-09-21 | University Of Electronic Science And Technology | Bearings-only target tracking method based on pseudo-linear maximum correlation entropy kalman filtering |
| CN117455960A (en) * | 2023-12-22 | 2024-01-26 | 中国人民解放军海军工程大学 | Passive positioning filtering algorithm for airborne photoelectric system to ground under time-varying observation noise condition |
-
2024
- 2024-04-03 CN CN202410403488.6A patent/CN117990112B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120022784A1 (en) * | 2010-06-25 | 2012-01-26 | Thales | Navigation filter for a navigation system using terrain correlation |
| JP2013061309A (en) * | 2011-09-15 | 2013-04-04 | Yamaha Corp | Kalman filter, state estimation device, method for controlling kalman filter, and control program of kalman filter |
| JP2015219714A (en) * | 2014-05-16 | 2015-12-07 | 日本電信電話株式会社 | State estimation device, method, and program |
| US20200132775A1 (en) * | 2017-06-14 | 2020-04-30 | Mitsubishi Electric Corporation | State estimation device |
| CN109990786A (en) * | 2019-02-28 | 2019-07-09 | 深圳大学 | Maneuvering target tracking method and device |
| CN110196443A (en) * | 2019-06-06 | 2019-09-03 | 中国人民解放军战略支援部队信息工程大学 | A kind of fault-tolerance combined navigation method and system of aircraft |
| CN110728716A (en) * | 2019-09-04 | 2020-01-24 | 深圳市道通智能航空技术有限公司 | Calibration method and device and aircraft |
| CN111896008A (en) * | 2020-08-20 | 2020-11-06 | 哈尔滨工程大学 | Improved robust unscented Kalman filtering integrated navigation method |
| US20220065587A1 (en) * | 2020-08-31 | 2022-03-03 | Raytheon Company | System and method of hypersonic object tracking |
| WO2022126397A1 (en) * | 2020-12-15 | 2022-06-23 | 深圳市大疆创新科技有限公司 | Data fusion method and device for sensor, and storage medium |
| CN112683274A (en) * | 2020-12-22 | 2021-04-20 | 西安因诺航空科技有限公司 | Unmanned aerial vehicle integrated navigation method and system based on unscented Kalman filtering |
| WO2022205526A1 (en) * | 2021-04-01 | 2022-10-06 | 江苏科技大学 | Dynamic positioning information fusion method for unmanned underwater vehicle cluster |
| US20230297642A1 (en) * | 2022-03-18 | 2023-09-21 | University Of Electronic Science And Technology | Bearings-only target tracking method based on pseudo-linear maximum correlation entropy kalman filtering |
| CN115981148A (en) * | 2022-12-07 | 2023-04-18 | 哈尔滨工业大学 | Unmanned aerial vehicle ground moving target tracking method |
| CN116520380A (en) * | 2023-04-28 | 2023-08-01 | 南京邮电大学 | Dynamic target combined positioning method based on self-adaptive Kalman filtering |
| CN117455960A (en) * | 2023-12-22 | 2024-01-26 | 中国人民解放军海军工程大学 | Passive positioning filtering algorithm for airborne photoelectric system to ground under time-varying observation noise condition |
Non-Patent Citations (8)
| Title |
|---|
| WANG, SY (WANG, SIYING) [1] ; THEN, A (THEN, ALEXANDER) [2] ; HERSCHEL, R (HERSCHEL, REINHOLD): "UAV Tracking based on Unscented Kalman Filter for Sense and Avoid Applications", 21ST INTERNATIONAL RADAR SYMPOSIUM, 31 October 2020 (2020-10-31), pages 250 - 255, XP033858151, DOI: 10.23919/IRS48640.2020.9253789 * |
| 卢航;郝顺义;彭志颖;黄国荣;: "基于MCC的鲁棒高阶CKF在组合导航中的应用", 计算机工程与应用, no. 01, 1 January 2020 (2020-01-01) * |
| 叶晨;崔双喜;: "一种强跟踪UKF及其在GPS/SINS深组合导航中的应用", 导弹与航天运载技术, no. 02, 10 April 2018 (2018-04-10) * |
| 张召: "高超声速机动目标跟踪模型与状态估计算法研究", 航空宇航科学与技术, 17 December 2020 (2020-12-17) * |
| 彭美康;郭蕴华;汪敬东;牟军敏;胡义;: "基于鲁棒容积卡尔曼滤波的自适应目标跟踪算法", 控制理论与应用, no. 04, 31 December 2020 (2020-12-31), pages 100 - 107 * |
| 李迟;李书臣;曹江涛;: "改进UKF算法在无人机姿态解算中的应用", 自动化技术与应用, no. 10, 25 October 2017 (2017-10-25), pages 57 - 60 * |
| 许万, 程兆, 夏瑞东, 等: "一种基于动态残差的自适应鲁棒无迹卡尔曼滤波器定位算法", 中国机械工程, vol. 34, no. 21, 30 November 2023 (2023-11-30), pages 2607 - 2614 * |
| 许剑锟, 金国栋, 谭力宁, 等: "基于平滑变结构-卡尔曼滤波的目标定位方法", 电光与控制, vol. 29, no. 3, 29 March 2022 (2022-03-29), pages 6 - 10 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117990112B (en) | 2024-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110243358B (en) | Multi-source fusion unmanned vehicle indoor and outdoor positioning method and system | |
| AU2012328156B2 (en) | Identification and analysis of aircraft landing sites | |
| Kong et al. | Autonomous landing of an UAV with a ground-based actuated infrared stereo vision system | |
| CN106871927B (en) | Unmanned aerial vehicle photoelectric pod installation error calibration method | |
| CN117455960B (en) | Passive positioning filtering method for airborne photoelectric system to ground under time-varying observation noise condition | |
| CN111238469B (en) | Unmanned aerial vehicle formation relative navigation method based on inertia/data chain | |
| Sabatini et al. | Navigation and guidance system architectures for small unmanned aircraft applications | |
| Goppert et al. | Invariant Kalman filter application to optical flow based visual odometry for UAVs | |
| CN117308938B (en) | Inertial navigation north-seeking convergence error rapid compensation method based on multiple laser ranging | |
| Mostafa et al. | Optical flow based approach for vision aided inertial navigation using regression trees | |
| CN117990112B (en) | Unmanned aerial vehicle photoelectric platform target positioning method based on robust unscented Kalman filtering | |
| CN115618749B (en) | Error compensation method for real-time positioning of large unmanned aerial vehicle | |
| CN115388890A (en) | Visual sense-based multi-unmanned aerial vehicle cooperative ground target positioning method | |
| CN112268558B (en) | Active target positioning and speed measuring method based on unmanned aerial vehicle autonomous photoelectric platform | |
| CN113189585A (en) | Motion error compensation algorithm based on unmanned aerial vehicle bistatic SAR system | |
| Popov et al. | UAV navigation on the basis of video sequences registered by onboard camera | |
| Duo et al. | Uav autonomous navigation system for gnss invalidation | |
| Ishii et al. | Autonomous UAV flight using the Total Station Navigation System in Non-GNSS Environments | |
| Wu et al. | Integrated navigation algorithm based on vision-inertial extended Kalman filter for low-cost unmanned aerial vehicle | |
| Xu et al. | An airborne LiDAR/INS integrated navigation algorithm based on fuzzy controlled SIFT | |
| Zahedi et al. | Neural network based approaches to mobile target localization and tracking using unmanned aerial vehicles | |
| Zhang et al. | Robust Autonomous Navigation Method for High-Precision UAV Based on Inertial/Machine Vision Fusion | |
| Sabatini et al. | Design and integration of vision based sensors for unmanned aerial vehicles navigation and guidance | |
| CN115307638A (en) | Hopfield network-based optimal path planning method and system for large-scale building aerial survey | |
| Abousleiman et al. | Statistical Algorithm for Attitude Estimation from Real-Time Aerial Video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |