CN106777489B - Tracking state modeling simulation method for unmanned aerial vehicle-mounted photoelectric stable turntable - Google Patents
Tracking state modeling simulation method for unmanned aerial vehicle-mounted photoelectric stable turntable Download PDFInfo
- Publication number
- CN106777489B CN106777489B CN201611029070.5A CN201611029070A CN106777489B CN 106777489 B CN106777489 B CN 106777489B CN 201611029070 A CN201611029070 A CN 201611029070A CN 106777489 B CN106777489 B CN 106777489B
- Authority
- CN
- China
- Prior art keywords
- aerial vehicle
- unmanned aerial
- axis
- angle
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000004088 simulation Methods 0.000 title claims abstract description 28
- 230000000007 visual effect Effects 0.000 claims abstract description 58
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 47
- 230000006641 stabilisation Effects 0.000 claims abstract description 7
- 238000011105 stabilization Methods 0.000 claims abstract description 7
- 238000012795 verification Methods 0.000 claims abstract description 5
- 238000004364 calculation method Methods 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 27
- 230000014509 gene expression Effects 0.000 claims description 18
- 230000003287 optical effect Effects 0.000 claims description 18
- 238000005096 rolling process Methods 0.000 claims description 17
- 230000009466 transformation Effects 0.000 claims description 13
- 230000005484 gravity Effects 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 5
- 239000000126 substance Substances 0.000 claims description 2
- HOWHQWFXSLOJEF-MGZLOUMQSA-N systemin Chemical compound NCCCC[C@H](N)C(=O)N[C@@H](CCSC)C(=O)N[C@@H](CCC(N)=O)C(=O)N[C@@H]([C@@H](C)O)C(=O)N[C@@H](CC(O)=O)C(=O)OC(=O)[C@@H]1CCCN1C(=O)[C@H]1N(C(=O)[C@H](CC(O)=O)NC(=O)[C@H](CCCN=C(N)N)NC(=O)[C@H](CCCCN)NC(=O)[C@H](CO)NC(=O)[C@H]2N(CCC2)C(=O)[C@H]2N(CCC2)C(=O)[C@H](CCCCN)NC(=O)[C@H](CO)NC(=O)[C@H](CCC(N)=O)NC(=O)[C@@H](NC(=O)[C@H](C)N)C(C)C)CCC1 HOWHQWFXSLOJEF-MGZLOUMQSA-N 0.000 claims description 2
- 108010050014 systemin Proteins 0.000 claims description 2
- 125000004432 carbon atom Chemical group C* 0.000 claims 1
- 238000012549 training Methods 0.000 abstract description 8
- 238000013461 design Methods 0.000 abstract description 6
- 238000012360 testing method Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000009434 installation Methods 0.000 description 4
- 230000000087 stabilizing effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000005094 computer simulation Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B9/00—Simulators for teaching or training purposes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Geometry (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Business, Economics & Management (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention discloses a tracking state modeling simulation method of an unmanned aerial vehicle-mounted photoelectric stabilization turntable, which is used for realizing a tracking state function of the photoelectric turntable based on digital modeling algorithm simulation, carrying three-dimensional visual scene virtual simulation training of photoelectric stabilization turntable unmanned aerial vehicle equipment, or carrying theoretical verification of a novel photoelectric stabilization turntable, and mainly comprises two parts of realizing photoelectric turntable functional modeling and photoelectric turntable nonlinear direction line tracking algorithm. The invention constructs a two-degree-of-freedom functional model of the photoelectric turntable, designs and realizes a nonlinear direction line tracking algorithm, well solves the problems of functional modeling and tracking algorithm modeling of the photoelectric turntable in three-dimensional visual simulation test, and has good engineering usability.
Description
Technical Field
The invention relates to the field of unmanned aerial vehicle platform simulation algorithms, in particular to a tracking state modeling simulation method for an unmanned aerial vehicle-mounted photoelectric stable turntable.
Background
The unmanned aerial vehicle carries a day and night photoelectric stabilization turntable (photoelectric turntable for short) which is a bearing mechanism of a visible light camera, a forward-looking infrared instrument, a laser distance measuring/indicating device and an optical stabilization aiming device, and has high-technology precision equipment for isolating the influence of the angular motion of the unmanned aerial vehicle on the bearing device and realizing the posture control function in a self-adaptive manner. Unmanned aerial vehicle army officers and soldiers find in equipping the flight training process, and the tracking operation process to the ground target when photoelectric rotary table is in tracking attitude, because the influence of factors such as weather visibility, the higher flight height of unmanned aerial vehicle, the operation process is very complicated, and the technical proficiency requirement of operative employee is high, and is higher to the degree of dependence of dress flight process. Because unmanned aerial vehicle flight airspace is wider, and the airspace coordination is comparatively complicated, the condition that the unmanned aerial vehicle army carries out the real dress flight receives the restriction of a great deal of factor, and moreover, the life that unmanned aerial vehicle equipped is limited, does not allow to carry out the real dress flight training frequently, consequently, the urgent need is based on the alternative training equipment of three-dimensional visual simulation technique, solves the training difficult problem that unmanned aerial vehicle army video reconnaissance target trailed. The full-digital simulation modeling research aiming at the three-dimensional visual application of the unmanned airborne photoelectric turntable hardware equipment is not available at home and abroad.
Disclosure of Invention
The invention aims to provide a tracking state modeling simulation method of an unmanned aerial vehicle-mounted photoelectric stable turntable, which aims to solve the problem that high-fidelity simulation of unmanned aerial vehicle equipment training under the software simulation condition in the prior art is difficult to realize.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
the tracking state modeling simulation method of the unmanned aerial vehicle-mounted photoelectric stabilized rotary table is characterized by comprising the following steps of:
firstly, simplifying the function of an unmanned aerial vehicle-mounted photoelectric stabilization turntable into a two-axis two-frame structure consisting of an inner ring assembly, an outer ring assembly and a support, wherein the inner ring assembly is provided with a viewpoint, the inner ring assembly is arranged on the outer ring assembly through a rotating shaft and can rotate around the outer ring shaft in an azimuth direction along with the outer ring assembly, the inner ring assembly rotates around the rotating shaft relative to the outer ring, the rotating angle of the inner ring assembly is marked as a model height angle, the rotating shaft of the outer ring assembly is fixed on the support and can rotate around the outer ring shaft relative to the support in the azimuth direction, the rotating angle is marked as a model azimuth angle, and the support is a mounting and fixing device and can be fixed on;
then, establishing an unmanned aerial vehicle N system coordinate, wherein the origin of the coordinate is set on the mass center of the unmanned aerial vehicle, and X isnAxis pointing to geographical north, ZnThe axis being in the direction of gravity, YnThe axis is eastward;
establishing a coordinate origin of a flight path S system at the mass center of the unmanned aerial vehicle, XsPlanned course of shaft and unmanned aerial vehicleIn the same direction, ZsThe axis being in the direction of gravity, YsAxis and XsAxis and ZsAxes are in right-hand rule relationship;
establishing coordinates of an unmanned plane U system, fixedly connecting the unmanned plane U system with an unmanned plane body, and obtaining an origin of coordinates OuIs the center of mass of the unmanned aerial vehicle, XuThe axis is the nose direction of the unmanned aerial vehicle, ZuThe axis being perpendicular to the plane of the fuselage and pointing downwards, YuAxis and XuAxis and ZuThe axes form a right-hand rule relationship; yaw angle
For unmanned plane U system XuAxial projection is at unmanned aerial vehicle N X of beingnOnYnAn azimuth on the plane; pitch angle
X for unmanned plane U systemuOuYuX of plane and unmanned aerial vehicle N systemnOnYnThe included angle of the plane is positive upwards; roll angle
X for unmanned plane U systemuOuZuX of plane and unmanned aerial vehicle N systemnOnZnThe right rotation of the included angle of the plane is positive;
establishing a coordinate system of the photoelectric turntable, wherein the coordinate system of the photoelectric turntable comprises a base coordinate system, an azimuth ring A system, a high-low ring F system, a rolling ring R system, a base B system and a coordinate origin ObIs the center of the rotating shaft, ZbThe axis being directed by the lens towards the target, XbAxis, YbAxes parallel to X of unmanned aerial vehicle U system respectivelyuAxis, YuA shaft; the azimuth ring A is fixedly connected with the azimuth ring of the photoelectric turntable and can only wind Z relative to the base BbThe shaft rotates to generate an azimuth angle theta of the photoelectric turntablea(ii) a High and low rings F are fixedly connected with high and low rings YfThe axis along the high-low ring axis and the azimuth ring A are YaThe axes are in the same direction; a high-low ring F system, which is an azimuth ring A wound around YaHigh and low angle theta of shaft rotationfAnd then obtaining; rolling ring R and rollingRing fixed connection, XrThe axis is along the transverse rolling ring axis and is parallel to the high-low ring XfThe rolling rings R are high and low rings F in the same axial direction and wound around XfTransverse rolling angle theta of shaft rotationrObtaining;
establishing the coordinates of an image plane M system, and establishing the origin O of the coordinates of the image plane M systemmAs principal point, ZmThe axis is parallel to the optical axis and points to the target, when the three attitude angles of the photoelectric turntable are all zero, the image plane M is YmAxis perpendicular to flight direction to the right, XmAxis perpendicular to YmIn the axial direction, the Z coordinate of the image plane M system is the focal length f;
establishing C system coordinates of a camera and a C system coordinate origin O of the cameracAt the optical center of the camera, i.e. the image principal point, XcX with axes parallel to the M system of the image planemAxis, YcY with axes parallel to the image plane M systemmAxis, ZcX with axes parallel to the M system of the image planemA shaft;
establishing coordinates of a ground station K system and an unmanned aerial vehicle K system, wherein the origin of the ground station K system and the origin of the unmanned aerial vehicle K system are respectively the center of mass of the ground station and the center of mass of the unmanned aerial vehicle, and X iskAxial east, YkAxial north, ZkThe axis is opposite to gravity;
let the tracked target point T be [ x ] in the K system coordinate of the ground stationt yt zt]TThe coordinate of the unmanned plane instantaneous position U in the ground station K system is [ x ]u yu zu]TConnecting the instantaneous position of the unmanned aerial vehicle to the tracked target in the ground station K system
In order to track a direction line, the functional model tracks the real-time attitude angles (alpha, beta) of a target, wherein alpha and beta are a model azimuth angle and a model elevation angle respectively, and the intersection point of a model visual axis and the ground is a principal point OcC system coordinate [ 00L ] of camera]TWherein L is the visual axis slope distance;
and finally, carrying out nonlinear direction line tracking calculation, wherein the nonlinear direction line tracking calculation comprises three steps of calculating the direction cosine of a tracking direction line, calculating the direction cosine of an unmanned aerial vehicle N system of a model visual axis and calculating the attitude angle of a model by DFP, and the specific process comprises the following steps:
(1) resolving the cosine of the direction of the tracking direction line:
the method comprises three calculation steps of tracking direction line unmanned aerial vehicle K system coordinate calculation, tracking direction line direction cosine calculation and tracking direction line coordinate transformation;
calculating tracking direction line
The coordinate component in the K system of the unmanned aerial vehicle is as shown in formula (1):
then track the direction line
Direction cosine of
The coordinate component expression of the unmanned plane K system is shown as formula (2):
will track the direction line
Direction cosine of
Transform into unmanned aerial vehicle N by unmanned aerial vehicle K system is system, and the transform formula is shown as equation (3):
(3) and DFP resolving model attitude angle:
the visual axis of the model is always coincident with the tracking direction line, so that the tracking of the visual angle of the functional model to the ground interested target is realized, namely the cosine of the visual axis direction of the model is ensured
Direction cosine of tracking direction line
Under the condition of real-time equality, the cosine of the visual axis direction of the known model
Reversely resolving the azimuth angle and the elevation angle of the model, which is a basic idea of a nonlinear direction line tracking algorithm;
during three-dimensional visual simulation or photoelectric turntable theoretical verification, the course angle, attitude angle, unmanned aerial vehicle position and ground target position of the unmanned aerial vehicle are known quantities, the direction cosine of a tracking direction line can be calculated, the direction cosine is given to a model visual axis direction line, the calculation of a model azimuth angle and a model elevation angle is a reverse calculation process for the model visual axis, the reverse calculation process is a typical nonlinear calculation process, the calculation is difficult to be directly performed by an analytic expression, a numerical iteration method is required to be used for solving, and the cost function J (·) expression of numerical iteration calculation is as shown in formula (7):
the model attitude angle in equation (7) is solved by using a quasi-Newton numerical iterative algorithmThe method is realized by using a DFP algorithm, and in order to achieve the aim of improving the efficiency of the algorithm by fast convergence, a model attitude angle DFP iterative algorithm needs a proper initial value, so that the model attitude angle at the previous moment is used as the iterative initial value at the current moment, and the correct solution is not deviated from the previous moment too far due to the influence of disturbance, and then the direction cosine of the tracking direction line at the current moment is used
Giving the current model the cosine of the visual axis direction
And taking the current model attitude angle as an iteration initial value, taking the current unmanned aerial vehicle attitude angle and the unmanned aerial vehicle course angle as known parameters, and iteratively calculating the model attitude angle, namely the model azimuth angle and the model altitude angle, by using a quasi-Newton numerical value iteration algorithm.
Compared with the prior art, the invention has the beneficial effects that:
the tracking state of the photoelectric turntable is an important working mode when the unmanned aerial vehicle executes a combat task, and is mainly used for completing the visual tracking task of a borne video camera/infrared instrument/laser indicator/optical sighting device visual axis to a ground target. The invention mainly solves the problem of digital function modeling of the photoelectric turntable on ground target visual tracking, strives to avoid purchasing physical equipment of the photoelectric turntable, and realizes the video/image/optical tracking task of the photoelectric turntable hardware on the ground target by only using computer software simulation. The technical problem to be solved by the invention comprises the steps of realizing the abstract modeling of the operation level function of the photoelectric turntable and designing a tracking state simulation algorithm, and completing the high-fidelity simulation realization of the tracking state function of the photoelectric turntable. The photoelectric turntable operation layer function abstract modeling emphasizes abstract modeling from the photoelectric turntable task execution function layer, and the design realizes the operation freedom degree, the bearing equipment installation position (the visual angle position of a camera/a front-view infrared instrument/a laser indicator/an optical aiming device), the equipment response parameter, the installation position and the like which are completely consistent with the actual installation. The invention relates to modeling of a tracking state software simulation algorithm of a photoelectric turntable, which is the core content of the invention, aims to use an intelligent calculation method and a digital modeling algorithm, solves the function simulation of the tracking state of hardware of the photoelectric turntable, aims to replace the video/image/optical tracking problem of the hardware of the photoelectric turntable on a ground target in a three-dimensional simulation training software and hardware environment, and solves the high-fidelity simulation realization problem of unmanned aerial vehicle equipment training under the software simulation condition.
The working principle of the invention is as follows: a complex mechanical servo structure, a gyroscope stabilizing loop and an image controller self-adaptive control loop of a practical photoelectric turntable are simplified and abstracted into a two-degree-of-freedom functional model consisting of a viewpoint (visual axis), an inner ring and an outer ring, the azimuth angle and the elevation angle of the functional model are solved in real time by using a quasi-Newton data value iterative algorithm by timely following a tracking direction method with the model visual axis, the design of a nonlinear direction line tracking algorithm is realized, and the key technical problem of digital modeling simulation of a tracking state of the photoelectric turntable in three-dimensional visual simulation training and theoretical verification of the photoelectric turntable is solved.
The invention constructs a two-degree-of-freedom functional model of the photoelectric turntable, designs and realizes a nonlinear direction line tracking algorithm, well solves the problems of functional modeling and tracking algorithm modeling of the photoelectric turntable in three-dimensional visual simulation test, and has good engineering usability.
The invention well solves the problem of digital modeling simulation of the tracking state of the photoelectric turntable in a three-dimensional simulation environment. Software simulation tests show that the software simulation has high convergence speed and high calculation precision. After the model and the algorithm are loaded in a three-dimensional simulation scene for testing, the model can well realize control on the viewpoint, the fidelity of the effect of the model and the effect of a real photoelectric turntable is high, when the ground target is tracked by control, no matter what mode of the unmanned aerial vehicle is maneuvering, the center of the view field carried by the model is always aligned with the tracked target, the occupancy rate of computer resources is low, and the three-dimensional picture is smooth and stable.
Drawings
FIG. 1 is an abstract functional model illustration.
Fig. 2 is an unmanned aerial vehicle N-system coordinate diagram.
FIG. 3 is a graph of a track S system.
Fig. 4 is a U-system coordinate diagram of the drone.
Fig. 5 is a coordinate system diagram of the photoelectric turntable.
FIG. 6 is a graph of the image plane M and the camera C.
Fig. 7 is a flow chart of the nonlinear direction line tracking algorithm.
FIG. 8 is a block diagram of a process for solving the direction cosine of the tracking direction line.
Fig. 9 is a block diagram of a process of transforming coordinates from a camera C system to an unmanned aerial vehicle N system.
FIG. 10 is a DFP solution model attitude angle flow diagram.
FIG. 11 is a graph showing the variation of the azimuth angle of the model.
FIG. 12 is a graph showing the variation of the mode elevation angle.
FIG. 13 is a screenshot of a target tracking.
FIG. 14 is a second screenshot of a target tracking.
Fig. 15 is a screenshot three of a target tracking.
Fig. 16 is a screenshot four of a target tracking.
Fig. 17 is a screenshot five of a target tracking.
Detailed Description
The invention mainly comprises two parts, one is the functional modeling of the photoelectric turntable, and the other is the realization of the nonlinear direction line tracking algorithm of the photoelectric turntable.
Firstly, an abstract functional structure model of a photoelectric turntable:
the invention discloses a two-axis four-frame hardware-based photoelectric turntable, which is characterized in that a two-axis four-frame hardware is composed of a mechanical servo structure, a gyroscope stabilizing structure and an image controller self-adaptive control structure, is functionally simplified into a two-axis two-frame structure (abbreviated as a functional model), and consists of an inner ring assembly, an outer ring assembly and a support as shown in figure 1. The inner ring assembly is mounted with a viewpoint (comprising a camera/a front-view infrared instrument/a laser indicator/a sighting device/an optical sighting device, the virtual optical axis of the viewpoint is abbreviated as a model visual axis), the inner ring assembly rotating shaft is mounted on the outer ring assembly and can rotate around the outer ring shaft in azimuth along with the outer ring assembly, the inner ring assembly rotates around the rotating shaft relative to the outer ring, and the rotating angle is a high-low angle (abbreviated as a model high-low angle). The rotating shaft of the outer ring component is fixed on the support and can rotate around the outer ring shaft in an azimuth direction relative to the support, and the rotating angle of the rotating shaft is an azimuth angle (abbreviated as a model azimuth angle). The support is installation fixing device, can fix on motion or static platform such as unmanned aerial vehicle.
Design of nonlinear direction line tracking algorithm
Principle of algorithm
The tracking state of the photoelectric turntable is mainly used for controlling a carried camera/infrared instrument/laser indicator/optical sighting device to carry out video/image/optical tracking on a ground target, and the continuous alignment of the visual axis of the camera/infrared instrument/laser indicator/optical sighting device to the ground target is controlled and realized mainly by utilizing an image matching algorithm, a gyroscope measurement stabilizing circuit and an image controller self-adaptive control circuit.
The nonlinear direction line tracking algorithm is the core content of the invention, mainly uses intelligent calculation method and digital modeling algorithm to simulate the video/image/optical tracking control process of the ground target when the photoelectric turntable hardware works, namely the process of keeping the visual axis of the camera/front-view infrared instrument/laser indicator/optical aiming device carried by the photoelectric turntable continuously aligned with the ground interested target, and mainly comprises the functional simulation model and algorithm for constructing the self-adaptive control process of the gyroscope stabilizing mechanism, the mechanical servo mechanism and the image controller of the solid-mounted photoelectric turntable in the tracking state.
Specifically, the azimuth angle and the elevation angle calculated by the nonlinear direction line tracking algorithm are used for controlling the functional model provided by the invention so as to realize the real-time control of the visual axis of the model viewpoint. The basic principle of the nonlinear direction line tracking algorithm is that a pointing line (tracking direction line for short) of an airplane/target is set, so that the visual axis of a model is always consistent with the tracking direction line, and the azimuth angle and the elevation angle of the model are solved in a nonlinear mode through a quasi-Newton numerical iteration algorithm, so that the self-adaptive control of the visual angle of a functional model is realized.
Algorithm flow
As shown in FIG. 2, the unmanned plane N system, with its origin of coordinates set at the center of mass of the unmanned plane, XnAxis pointing to geographical north, ZnThe axis being in the direction of gravity, YnThe axis is eastward.
As shown in FIG. 3, the origin of coordinates of the track S system is at the center of mass of the UAV, XsAxis is in the same direction as the planned course of the unmanned aerial vehicle, ZsThe axis being in the direction of gravity, YsAxis and XsAxis and ZsThe axes are in right-hand rule relationship.
As shown in fig. 4, the unmanned plane U is fixedly connected with the unmanned plane body, and the origin of coordinates O thereofuIs the center of mass of the unmanned aerial vehicle, XuThe axis is the nose direction of the unmanned aerial vehicle, ZuThe axis being perpendicular to the plane of the fuselage and pointing downwards, YuAxis and XuAxis and ZuThe axes form a right-hand rule relationship; yaw angle
For unmanned plane U system XuAxial projection is at unmanned aerial vehicle N X of beingnOnYnAn azimuth on the plane; pitch angle
X for unmanned plane U systemuOuYuX of plane and unmanned aerial vehicle N systemnOnYnThe included angle of the plane is positive upwards; roll angle
X for unmanned plane U systemuOuZuX of plane and unmanned aerial vehicle N systemnOnZnThe right rotation of the included angle of the plane is positive;
the photoelectric turntable coordinate system comprises a base coordinate system, an azimuth ring A system, a high-low ring F system and a rolling ring R system. As shown in fig. 5, the base B system, origin of coordinates ObIs the center of the rotating shaft, ZbThe axis being directed by the lens towards the target, XbAxis, YbAxes parallel to X of unmanned aerial vehicle U system respectivelyuAxis, YuA shaft. The azimuth ring A is fixedly connected with the azimuth ring of the photoelectric turntable and can only wind Z relative to the base BbThe shaft rotates to generate an azimuth angle theta of the photoelectric turntablea. High and low rings F are fixedly connected with high and low rings YfThe axis along the high-low ring axis and the azimuth ring A are YaThe axes are in the same direction. The high-low ring F is a ring,is an azimuth circle A wound around YaHigh and low angle theta of shaft rotationfAnd obtaining the compound. The rolling ring R is fixedly connected with the rolling ring XrThe axis is along the transverse rolling ring axis and is parallel to the high-low ring XfThe rolling rings R are high and low rings F in the same axial direction and wound around XfTransverse rolling angle theta of shaft rotationrThus obtaining the product.
As shown in fig. 6, the image plane M system, the origin of coordinates OmAs principal point, ZmThe axis is parallel to the optical axis and points to the target, when the three attitude angles of the photoelectric turntable are all zero, the image plane M is YmAxis perpendicular to flight direction to the right, XmAxis perpendicular to YmAxially. The Z coordinate of the image plane M system is the focal length f.
As shown in fig. 6, the camera C has an origin O of coordinatescAt the optical center (i.e. image principal point) of the camera, XcX with axes parallel to the M system of the image planemAxis, YcY with axes parallel to the image plane M systemmAxis, ZcX with axes parallel to the M system of the image planemA shaft.
The ground station K system and the unmanned aerial vehicle K system have the coordinate origin of the mass center of the ground station and the mass center of the unmanned aerial vehicle respectively, and X iskAxial east, YkAxial north, ZkOpposite to gravity.
Let the coordinate of the tracked target T be [ x ]t yt zt]T(ground station K system) and the instantaneous position U of the unmanned aerial vehicle is [ x ]u yuzu]T(ground station K system) connecting the instantaneous position of the unmanned plane to the tracked target in the ground station K system
For tracking a direction line, real-time attitude angles (alpha, beta) of the functional model for tracking the target (alpha, beta are respectively a model azimuth angle and a model elevation angle), and an intersection point (a principal point) O of a model visual axis and the groundcC system coordinate [ 00L ] of camera]T(L is the visual axis slope distance).
The nonlinear direction line tracking algorithm provided by the invention has a calculation flow as shown in fig. 7, and mainly comprises three steps of calculating the direction cosine of a tracking direction line, calculating the direction cosine of an unmanned aerial vehicle N system of a model visual axis and calculating the model attitude angle by DFP.
(1) Solving the cosine of the direction of the tracking line
The calculating process is shown in figure 8 and mainly comprises three calculating steps of tracking direction line unmanned aerial vehicle K system coordinate calculating, tracking direction line direction cosine calculating and tracking direction line coordinate transformation.
Calculating tracking direction line
Coordinate components in the drone K system.
Then track the direction line
Direction cosine of
The coordinate component expression of the unmanned plane K system is as follows:
will track the direction line
Direction cosine of
Transform into unmanned aerial vehicle N by unmanned aerial vehicle K system, the transform is as follows:
(2) unmanned aerial vehicle N system direction cosine for resolving model visual axis
Based on the principle of coordinate transformation (the transformation process is shown in FIG. 9), the principal point of the earth OcThe coordinate transformation from the camera C system to the drone N system is as follows.
Wherein the content of the first and second substances,
and
the rotation transformation matrix from the unmanned plane N system to the track S system, from the track S system to the unmanned plane U system, from the base B system to the azimuth ring A system, and from the azimuth ring A system to the high-low ring F system respectively corresponds to the course angle phi of the unmanned planehxUnmanned aerial vehicle attitude angle (yaw angle)
Pitch angle
And roll angle
) Attitude angle (azimuth angle theta) of photoelectric turntablefwHigh and low angle thetagd) The equal input angle parameters have expressions shown in formulas (5.1) - (5.4).
Wherein M isx、My、MzRespectively, basic rotation matrices around the X-axis, Y-axis and Z-axis,
orientation cosine of unmanned aerial vehicle N system of model visual axis
The expression is as follows.
Wherein x in the above formulan、ynAnd znThe expressions are shown as formulas (6.1), (6.2) and (6.3).
Then
Namely, the direction cosine of the unmanned aerial vehicle N system of the model visual axis at the moment.
(3) DFP (design flow Pattern) calculation model attitude angle
The model visual axis is always coincident with the tracking direction line, so that the tracking of the functional model visual angle to the ground interested target is realized, namely, the cosine of the model visual axis direction is ensured
Direction cosine of tracking direction line
Under the condition of real-time equality, the cosine of the visual axis direction of the known model
And reversely solving the azimuth angle and the elevation angle of the model, which is the basic idea of the nonlinear direction line tracking algorithm. The solution flow is shown in fig. 10.
During three-dimensional visual simulation or photoelectric turntable theoretical verification, the course angle, attitude angle, unmanned aerial vehicle position and ground target position of the unmanned aerial vehicle are known quantities, the direction cosine of a tracking direction line can be calculated, the direction cosine is given to a model visual axis direction line, the calculation process according to the model visual axis of 4.2 sections can be known, the calculation process of the model azimuth angle and the model elevation angle is the reverse calculation process of the model visual axis, the calculation process is a typical nonlinear calculation process, the calculation process is difficult to directly use an analytic expression, a numerical iteration method is required to be used for solving, and the cost function J (·) expression of the numerical iteration calculation is as follows:
the method uses a quasi-Newton numerical iteration algorithm to solve the model attitude angle in the formula (7), and particularly uses a DFP algorithm to realize the method. Model attitude angle DFP iteration for fast convergence and improved algorithm efficiencyThe invention takes the model attitude angle of the previous moment as the iteration initial value of the current moment, which is based on the influence of disturbance to ensure that the correct solution does not deviate too far from the previous moment, and then uses the direction cosine of the tracking direction line of the current moment
Giving the current model the cosine of the visual axis direction
And (3) taking the current model attitude angle as an iteration initial value, taking the current unmanned aerial vehicle attitude angle and the unmanned aerial vehicle course angle as known parameters, and iteratively calculating the model attitude angle (the model azimuth angle and the model altitude angle) by using a quasi-Newton numerical iteration algorithm.
In order to verify the correctness of the tracking state simulation modeling technology of the photoelectric turntable, the method simulates the airplane to fly in a large circle around a certain point on the ground as a center (the course angle of the airplane is changed greatly), the model viewpoint tracks the central point of the large circle, and the real-time change of the attitude angle of the model is solved by utilizing a nonlinear direction line tracking algorithm, namely the adjustment and change process of the azimuth angle of the model and the elevation angle of the model is used for verifying the correctness of the functional model and the tracking algorithm provided by the patent.
The specific parameters tested were as follows: function model initial attitude angle [0 °, 0 ° ]]TSimulating the initial attitude angle of the unmanned plane [0 degrees, 0 degrees and 0 degrees ]]TThe initial heading of the simulated unmanned aerial vehicle is due north, the flight height is 1000 meters, the flight speed is 190 kilometers per hour, and the radius of the circular flight large circle is 1.5 kilometers. The model azimuth angle and the model elevation angle variation curve solved by the nonlinear direction line tracking algorithm are shown in fig. 11 and fig. 12.
The simulation result shows that when the aircraft makes large circular motion and the center of the circle is set as a video tracking target, as the course angle of the unmanned aerial vehicle makes continuous uniform rate (uniform rate turning motion of the unmanned aerial vehicle) change, in order to realize effective tracking of the target, the nonlinear direction line tracking algorithm adjusts the posture of the functional model in real time, the azimuth angle of the model is linearly changed, the uniform rate change of the course angle of the unmanned aerial vehicle is compensated, and the altitude angle of the model is nonlinearly changed; the break points shown in the figure are numerical jumps of the model azimuth angle/model elevation angle at 0 degree or 360 degrees, and are not true break points. It can be seen that the functional model and the nonlinear direction line tracking algorithm provided by the invention are completely correct in theory.
The functional model is loaded to a three-dimensional simulation scene, the nonlinear direction line tracking algorithm is used for controlling the view angle of the three-dimensional scene after software engineering, the software engineering test result is captured as shown in fig. 13-17, video tracking simulation of a given target in the scene is completely realized, no matter what motion mode the unmanned aerial vehicle does or no matter whether the unmanned aerial vehicle moves to any place, as long as the view field is visible, the center of the view field is always aligned with the tracked target, the occupancy rate of computer resources is low, the three-dimensional picture is smooth and stable, the technical theory is correct, and the technology is proved to have better engineering usability.
Claims (1)
1. The tracking state modeling simulation method of the unmanned aerial vehicle-mounted photoelectric stabilized rotary table is characterized by comprising the following steps of:
firstly, simplifying the function of an unmanned aerial vehicle-mounted photoelectric stabilization turntable into a two-axis two-frame structure consisting of an inner ring assembly, an outer ring assembly and a support, wherein the inner ring assembly is provided with a viewpoint, the inner ring assembly is arranged on the outer ring assembly through a rotating shaft and can rotate around the outer ring shaft in an azimuth direction along with the outer ring assembly, the inner ring assembly rotates around the rotating shaft relative to the outer ring, the rotating angle of the inner ring assembly is marked as a model height angle, the rotating shaft of the outer ring assembly is fixed on the support and can rotate around the outer ring shaft relative to the support in the azimuth direction, the rotating angle is marked as a model azimuth angle, and the support is a mounting and fixing device and;
then, establishing an unmanned aerial vehicle N system coordinate, wherein the origin of the coordinate is set on the mass center of the unmanned aerial vehicle, and X isnAxis pointing to geographical north, ZnThe axis being in the direction of gravity, YnThe axis is eastward;
establishing a coordinate origin of a flight path S system at the mass center of the unmanned aerial vehicle, XsAxis is in the same direction as the planned course of the unmanned aerial vehicle, ZsThe axis being in the direction of gravity, YsAxis and XsAxis and ZsAxes are in right-hand rule relationship;
establishing coordinates of an unmanned plane U system, fixedly connecting the unmanned plane U system with an unmanned plane body, and obtaining an origin of coordinates OuIs the center of mass of the unmanned aerial vehicle, XuThe axis is the nose direction of the unmanned aerial vehicle, ZuThe axis being perpendicular to the plane of the fuselage and pointing downwards, YuAxis and XuAxis and ZuThe axes form a right-hand rule relationship; yaw angle
For unmanned plane U system XuAxial projection is at unmanned aerial vehicle N X of beingnOnYnAn azimuth on the plane; pitch angle
X for unmanned plane U systemuOuYuX of plane and unmanned aerial vehicle N systemnOnYnThe included angle of the plane is positive upwards; roll angle
X for unmanned plane U systemuOuZuX of plane and unmanned aerial vehicle N systemnOnZnThe right rotation of the included angle of the plane is positive;
establishing a coordinate system of the photoelectric turntable, wherein the coordinate system of the photoelectric turntable comprises a base coordinate system, an azimuth ring A system, a high-low ring F system, a rolling ring R system, a base B system and a coordinate origin ObIs the center of the rotating shaft, ZbThe axis being directed by the lens towards the target, XbAxis, YbAxes parallel to X of unmanned aerial vehicle U system respectivelyuAxis, YuA shaft; the azimuth ring A is fixedly connected with the azimuth ring of the photoelectric turntable and can only wind Z relative to the base BbThe shaft rotates to generate an azimuth angle theta of the photoelectric turntablea(ii) a High and low rings F are fixedly connected with high and low rings YfThe axis along the high-low ring axis and the azimuth ring A are YaThe axes are in the same direction; a high-low ring F system, which is an azimuth ring A wound around YaHigh and low angle theta of shaft rotationfAnd then obtaining; the rolling ring R is fixedly connected with the rolling ring XrThe shaft is parallel to the transverse rolling ring shaftLow ring XfThe rolling rings R are high and low rings F in the same axial direction and wound around XfTransverse rolling angle theta of shaft rotationrObtaining;
establishing the coordinates of an image plane M system, and establishing the origin O of the coordinates of the image plane M systemmAs principal point, ZmThe axis is parallel to the optical axis and points to the target, when the three attitude angles of the photoelectric turntable are all zero, the image plane M is YmAxis perpendicular to flight direction to the right, XmAxis perpendicular to YmIn the axial direction, the Z coordinate of the image plane M system is the focal length f;
establishing C system coordinates of a camera and a C system coordinate origin O of the cameracAt the optical center of the camera, i.e. the image principal point, XcX with axes parallel to the M system of the image planemAxis, YcY with axes parallel to the image plane M systemmAxis, ZcX with axes parallel to the M system of the image planemA shaft;
establishing coordinates of a ground station K system and an unmanned aerial vehicle K system, wherein the origin of the ground station K system and the origin of the unmanned aerial vehicle K system are respectively the center of mass of the ground station and the center of mass of the unmanned aerial vehicle, and X iskAxial east, YkAxial north, ZkOpposite to gravity;
let the tracked target point have the coordinate of [ x ] in the K system of the ground stationt yt zt]TThe coordinate of the instantaneous position of the unmanned aerial vehicle in the ground station K system is [ x ]u yu zu]TConnecting the instantaneous position of the unmanned aerial vehicle to the tracked target in the ground station K system
In order to track a direction line, the functional model tracks the real-time attitude angles (alpha, beta) of a target, wherein alpha and beta are a model azimuth angle and a model elevation angle respectively, and the intersection point of a model visual axis and the ground is a principal point OcC system coordinate [ 00L ] of camera]TWherein L is the visual axis slope distance;
and finally, carrying out nonlinear direction line tracking calculation, wherein the nonlinear direction line tracking calculation comprises three steps of calculating the direction cosine of a tracking direction line, calculating the direction cosine of an unmanned aerial vehicle N system of a model visual axis and calculating the attitude angle of a model by DFP, and the specific process comprises the following steps:
(1) resolving the cosine of the direction of the tracking direction line:
the method comprises three calculation steps of tracking direction line unmanned aerial vehicle K system coordinate calculation, tracking direction line direction cosine calculation and tracking direction line coordinate transformation;
calculating tracking direction line
The coordinate component in the K system of the unmanned aerial vehicle is as shown in formula (1):
then track the direction line
Direction cosine of
The coordinate component expression of the unmanned plane K system is shown as formula (2):
will track the direction line
Direction cosine of
Transform into unmanned aerial vehicle N by unmanned aerial vehicle K system is system, and the transform formula is shown as equation (3):
in the above formula, the first and second carbon atoms are,
to track the direction line
The unmanned aerial vehicle has N-system direction cosine with the components of rnx，rny，rnz，MenThe method is a rotation transformation matrix from an unmanned plane K system to an unmanned plane N system, and the expression formula is shown in formula (4):
(2) resolving the direction cosine of the unmanned aerial vehicle N system of the model visual axis:
based on the principle of coordinate transformation, the principal point of the earth OcThe coordinate transformation process from the camera C system to the unmanned aerial vehicle N system is shown as the formula (5):
wherein the content of the first and second substances,
and
the rotation transformation matrix from the unmanned plane N system to the track S system, from the track S system to the unmanned plane U system, from the base B system to the azimuth ring A system, and from the azimuth ring A system to the high-low ring F system respectively corresponds to the course angle phi of the unmanned planehxIncluding yaw angle
Pitch angle
And roll angle
Of unmanned aerial vehicle attitude angle, including azimuth angle thetafwHigh and low angle thetagdThe expression formula of the attitude angle of the photoelectric turntable is shown in formulas (5.1) to (5.4):
wherein M isx、My、MzRespectively, basic rotation matrices around the X-axis, Y-axis and Z-axis,
orientation cosine of unmanned aerial vehicle N system of model visual axis
The expression is shown in formula (6):
wherein x in the above formulan、ynAnd znThe expression is shown in formulas (6.1), (6.2) and (6.3):
then
Namely the direction cosine of the unmanned aerial vehicle N system of the model visual axis at the moment;
(3) and DFP resolving model attitude angle:
the visual axis of the model is always coincident with the tracking direction line, so that the tracking of the visual angle of the functional model to the ground interested target is realized, namely the cosine of the visual axis direction of the model is ensured
Direction cosine of tracking direction line
Under the condition of real-time equality, the cosine of the visual axis direction of the known model
Reversely resolving the azimuth angle and the elevation angle of the model, which is a basic idea of a nonlinear direction line tracking algorithm;
during three-dimensional visual simulation or photoelectric turntable theoretical verification, the course angle, attitude angle, unmanned aerial vehicle position and ground target position of the unmanned aerial vehicle are known quantities, the direction cosine of a tracking direction line can be calculated, the direction cosine is given to a model visual axis direction line, the calculation of a model azimuth angle and a model elevation angle is a reverse calculation process for the model visual axis, the reverse calculation process is a typical nonlinear calculation process, the calculation is difficult to be directly performed by an analytic expression, a numerical iteration method is required to be used for solving, and the cost function J (·) expression of numerical iteration calculation is as shown in formula (7):
the model attitude angle in the formula (7) is solved by using a quasi-Newton numerical iteration algorithm, the model attitude angle is specifically realized by using a DFP algorithm, in order to achieve the purpose of improving the efficiency of the algorithm by fast convergence, the model attitude angle DFP iteration algorithm needs a proper initial value, so the model attitude angle at the previous moment is used as the iteration initial value at the current moment, which is caused by the fact that the correct solution does not deviate from the previous moment too far based on the influence of disturbance, and then the direction cosine of the tracking direction line at the current moment is used
Giving the current model the cosine of the visual axis direction
And taking the current model attitude angle as an iteration initial value, taking the current unmanned aerial vehicle attitude angle and the unmanned aerial vehicle course angle as known parameters, and iteratively calculating the model attitude angle, namely the model azimuth angle and the model altitude angle, by using a quasi-Newton numerical value iteration algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611029070.5A CN106777489B (en) | 2016-11-22 | 2016-11-22 | Tracking state modeling simulation method for unmanned aerial vehicle-mounted photoelectric stable turntable |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611029070.5A CN106777489B (en) | 2016-11-22 | 2016-11-22 | Tracking state modeling simulation method for unmanned aerial vehicle-mounted photoelectric stable turntable |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106777489A CN106777489A (en) | 2017-05-31 |
| CN106777489B true CN106777489B (en) | 2021-04-06 |
Family
ID=58970651
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611029070.5A Expired - Fee Related CN106777489B (en) | 2016-11-22 | 2016-11-22 | Tracking state modeling simulation method for unmanned aerial vehicle-mounted photoelectric stable turntable |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106777489B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108536171B (en) * | 2018-03-21 | 2020-12-29 | 电子科技大学 | Path planning method for collaborative tracking of multiple unmanned aerial vehicles under multiple constraints |
| CN108733066B (en) * | 2018-05-07 | 2021-05-07 | 中国人民解放军国防科技大学 | Target tracking control method based on pod attitude feedback |
| CN110470304B (en) * | 2019-08-19 | 2021-04-20 | 西安因诺航空科技有限公司 | High-precision target positioning and speed measuring method based on unmanned aerial vehicle photoelectric platform |
| CN110856134B (en) * | 2019-10-16 | 2022-02-11 | 东南大学 | Large-scale wireless sensor network data collection method based on unmanned aerial vehicle |
| CN110672116B (en) * | 2019-11-04 | 2022-03-01 | 中国人民解放军空军工程大学 | Moving target consistency judgment method based on multi-observation whistle digital telescope |
| CN111142566A (en) * | 2019-11-05 | 2020-05-12 | 西安羚控电子科技有限公司 | Target tracking method for ship-based unmanned aerial vehicle in take-off and landing stages |
| CN113189890A (en) * | 2020-01-14 | 2021-07-30 | 北京京邦达贸易有限公司 | Simulation method and device for unmanned aerial vehicle target positioning |
| CN112882498B (en) * | 2021-01-12 | 2024-04-16 | 湖南宇正智能科技有限公司 | Triaxial photoelectric searching and tracking device and method with image rotation inhibiting function |
| CN112884798A (en) * | 2021-01-27 | 2021-06-01 | 湖北三江航天红峰控制有限公司 | Verification method of moving target tracking and aiming system |
| CN115134448A (en) * | 2022-06-07 | 2022-09-30 | 江西盈盛实业有限公司 | Backlight illuminating system for network telephone |
| CN115840288A (en) * | 2023-02-20 | 2023-03-24 | 长春理工大学 | Intelligent photoelectric turntable full-link image quality analysis method |
| CN116630374B (en) * | 2023-07-24 | 2023-09-19 | 贵州翰凯斯智能技术有限公司 | Visual tracking method, device, storage medium and equipment for target object |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2954778B2 (en) * | 1992-04-13 | 1999-09-27 | 日立ビアメカニクス株式会社 | Method for connecting line figure branch point of image input device |
| CN102520430B (en) * | 2011-12-29 | 2013-07-03 | 中国科学院长春光学精密机械与物理研究所 | Object locating device of small size airborne photoelectric tracking equipment |
| CN103604427B (en) * | 2013-12-10 | 2016-10-12 | 中国航天空气动力技术研究院 | UAS and method to ground moving target dynamic location |
| CN105354861A (en) * | 2015-09-28 | 2016-02-24 | 凯迈（洛阳）测控有限公司 | Target trajectory prediction based tracking method for airborne photo-electronic device |
-
2016
- 2016-11-22 CN CN201611029070.5A patent/CN106777489B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN106777489A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106777489B (en) | Tracking state modeling simulation method for unmanned aerial vehicle-mounted photoelectric stable turntable | |
| CN109359405B (en) | Air-space-ground integrated big data battlefield environment semi-physical simulation system | |
| CN102252673B (en) | Correction method for on-track aberration of star sensor | |
| CN106681170B (en) | Semi-physical guidance simulation method and system | |
| CN105786024B (en) | A kind of airborne photoelectric platform high precision tracking controller and its tracking and controlling method based on Compensation for Model Errors | |
| CN107390545A (en) | A kind of simulation training system of unmanned plane and its load | |
| CN105929836B (en) | Control method for quadrotor | |
| CN104932535B (en) | Utilize method of the analogue system to Air-borne Forward-looking Infra-Red Search Set closed loop test | |
| CN103604427A (en) | Unmanned aerial vehicle system and method for dynamically positioning ground moving target | |
| CN105151292A (en) | Distributive vectored thrust system | |
| CN107727101B (en) | Three-dimensional attitude information rapid resolving method based on dual-polarized light vector | |
| CN104501835A (en) | Ground test system and method targeting to space application isomerism IMU initial alignment | |
| CN117455960B (en) | Passive positioning filtering method for airborne photoelectric system to ground under time-varying observation noise condition | |
| CN115291536B (en) | Visual unmanned aerial vehicle tracking ground target semi-physical simulation platform verification method | |
| CN106529105B (en) | UAV system opto-electric stabilization turntable inertia state modeling and simulating method | |
| CN110220533A (en) | A kind of onboard electro-optical pod misalignment scaling method based on Transfer Alignment | |
| CN114594457A (en) | Device and method for testing dynamic three-dimensional data precision of multiband photoelectric system | |
| CN108489338B (en) | Infrared seeker line-of-sight angular rate precision testing method and system | |
| CN104898653A (en) | Flight control system | |
| CN111026139B (en) | Three-dimensional model posture adjustment control method based on flight track | |
| CN115541187A (en) | Airborne photoelectric system testing method under high dynamic condition | |
| Xu et al. | Simulation Modeling for Inertial Work-pattern of UAV Optoelectronic Gimbals | |
| CN112268558B (en) | Active target positioning and speed measuring method based on unmanned aerial vehicle autonomous photoelectric platform | |
| Yang et al. | Design, implementation, and verification of a low‐cost terminal guidance system for small fixed‐wing UAVs | |
| Yang et al. | An efficient simulation platform for testing and validating autonomous navigation algorithms for multi-rotor uavs based on unreal engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210406 |