CN112396626B - Sequential image population identification method for monitoring migration of waiting birds - Google Patents
Sequential image population identification method for monitoring migration of waiting birds Download PDFInfo
- Publication number
- CN112396626B CN112396626B CN202011553426.1A CN202011553426A CN112396626B CN 112396626 B CN112396626 B CN 112396626B CN 202011553426 A CN202011553426 A CN 202011553426A CN 112396626 B CN112396626 B CN 112396626B
- Authority
- CN
- China
- Prior art keywords
- frame
- bird
- target
- image
- centroid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/215—Motion-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/70—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in livestock or poultry
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
In order to solve the technical problem that the existing method cannot accurately detect the dynamic characteristics of bird population in the migration process, the invention provides a sequential image population identification method for monitoring the migration of a bird, which adopts an image sequence to track birds, dynamically matches continuous frame connected domains of the bird, effectively acquires bird population information, and simultaneously can accurately record the flight track, the number and the flight track.
Description
Technical field:
the invention relates to an image processing method, in particular to a sequential image population identification method for monitoring migration of a waiting bird, and belongs to the field of image processing.
The background technology is as follows:
recent researches show that the relationship exists between the migration of the bird and the avian influenza virus, so that the tracking of the migration track of the bird can provide early warning for influenza outbreak; in recent years, the blood color event of the migratory bird group is still existed by setting a net in the migratory course of the bird, and even if an ecological monitoring management station is established, the mode of organizing manpower is adopted to patrol the bird habitat, so that the bird habitat is difficult to be completely paid out on the net; knowledge of the migration time and the ecological laws of routes, migration time, population relations and the like of the waiting birds is very important for the protection effect of ecological resources and the prevention effect of influenza viruses.
The bird population identification, the statistics of the number and the tracking of the flight track are tedious and time-consuming work for bird researchers, and the workload of the bird researchers can be reduced to a certain extent by combining the prior scientific and technological means; by adopting an unmanned monitoring method, birds can be effectively monitored under the natural state without being influenced, and the monitoring modes of the existing acoustic device such as position discrimination, radar detection, satellite positioning tracking and the like can obtain more accurate monitoring results in specific application scenes; the acoustic monitoring method has low cost and long-term monitoring of bird activities, but the accuracy of the detection means is dependent on the accuracy of the bird song recognition method, and the specific number of bird migration cannot be counted; advanced radar equipment can obtain a large amount of information such as height, direction, speed and the like of birds during migration flight in a large range, but can only infer the variety of birds through the information such as individual size, wing vibration frequency and the like of the birds; the satellite tracking can track the migration track of birds with high precision, a monitor needs to place a satellite transmitter on an observed bird waiting individual body to acquire the position information of the bird waiting individual body, and the cost is high.
The image monitoring bird waiting migration adopts the current popular computer vision technology, so that the information of the monitored bird waiting population, the monitored bird waiting quantity, the monitored bird flight trajectories and the like can be more efficiently obtained, and a large amount of effective data are provided for researchers; in addition, by adopting an image monitoring mode, the bird image data of the unmanned area can be obtained, so that people can conveniently and deeply study birds; digital images have been widely studied in bird monitoring, but bird information acquired by the digital images is limited to single images or image sets obtained by sampling at fixed time intervals, so that the dynamic characteristics of bird populations in the migration process are ignored to a certain extent, and cannot be reflected by single two-dimensional images.
The invention comprises the following steps:
in order to solve the technical problem that the existing method cannot accurately detect the dynamic characteristics of bird population in the migration process, the invention provides a sequential image population identification method for monitoring the migration of a bird, which adopts an image sequence to track birds, dynamically matches continuous frame connected domains of the bird, effectively acquires bird population information, and simultaneously can accurately record the flight track, the number and the flight track.
The invention solves the technical problems by adopting the technical scheme that: a serial image population identification method for monitoring migration of a waiting bird is characterized by comprising the following steps:
step one, a monitoring method: monitoring a bird group by adopting high-frame-rate sequence images, identifying the types of bird targets, simultaneously carrying out real-time mark matching on bird target positions in continuous frames, recording bird flight tracks, and counting the number of birds according to mark numbers;
step two, the processing flow of the acquired image sequence is as follows: the observation sites of bird migration are selected from the places with a large number of waiting birds passing through in the migration period, such as sea islands far away from continents and mountain choke; obtaining continuous frame sequence images by adopting a high frame rate image sensor; the image acquisition process flow comprises the following steps: the method comprises the steps that a large target image is obtained for bird images in a threshold range of available discrimination types, a bird group image is obtained for images with the largest number of detected birds after filtering, and a frame sequence is obtained for all frame images from a first frame image to a last frame image which contain bird targets in sequence time arrangement;
step three, connected domain judgment:
judging the moving direction of the target, recording the mass center position of the target in the current frame, judging the moving range of the next frame according to the flying speed and the flying direction of birds, and searching the target in the moving range; according to the method, the connected domains where the same flying bird targets are located are matched according to the time sequence, so that the moving track of the flying bird can be marked, the newly-appearing targets in the field of view are marked, the shielding problem of the flying bird in different images can be effectively identified, and the number of flying bird groups can be counted accurately;
matching the continuous frame connected domain of the flying bird:
(1) After completing target segmentation for the current frame, marking the target position, numbering the target connected domain by 1, … …, i, and recording the centroid position;
(2) Judging the movement direction of the flying bird according to the continuous frame movement coordinates of the target, and predicting the movement direction and the distance of the target of the next frame;
(3) After the next frame of image is segmented, carrying out connected domain centroid position marking, screening and matching the connected domain according to the predicted direction in the step (2), carrying out Euclidean distance calculation with the screened connected domain centroid, and carrying out connected domain screening according to the predicted distance range in the step (2);
(4) After the step (3) is finished, if the number n of the connected domains matched by the frame and the previous frame is 1, the matching is considered to be correct, and the existing connected domain numbers 1, … … and i are not updated; if the number n of the connected domains matched with the previous frame is an integer greater than 1, judging that the previous frame has a shielding area, and sequentially updating the matched connected domain numbers to i, i+1, … … and i+n-1;
(5) Sequentially numbering the newly added connected domains of each frame to ensure that marked birds in the video sequence images are not repeatedly counted;
fourthly, determining the mass center of the connected domain and predicting the track:
(1) Connected domain centroid determination
Let the image function be f (x, y), then the p+q standard moment of the image is:
wherein p and q are positive integers;
the target centroid position is:
wherein: x, y are image coordinates; m and N respectively represent the height and width of the image;
representing the centroid coordinates of the image;
(2) Track prediction:
calculating a target centroid by adopting a geometric moment, calculating a target distance of an adjacent frame by adopting an Euclidean distance, and carrying out target range prediction of a next frame according to the current frame, the ith frame, the first two frames, the ith-1 frame and the ith-2 frame and cached data;
the frame sequence prediction model and the method are as follows: according to the i-2 th frame centroid coordinates
Coordinates of the centroid of the object with the i-1 th frame +.>
Predicting the possible centroid point position of the ith frame object +.>
Setting the point to be quasi-pre-determinedMeasuring point and calculating the deviation angle of the track and horizontal position>
Coordinates of centroid with object in frame i +.>
Calculating and predicting target position +.>
The prediction formula is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
in order to predict the deflection angle range in the horizontal direction, d is the predicted distance range, θ j D is the angle margin j Is the length allowance.
The beneficial effects of the invention are as follows: the birds are tracked by adopting the image sequence, and the continuous frame connected domain of the birds is dynamically matched, so that the bird population information can be effectively obtained, and meanwhile, the flight track, the number and the accuracy of the bird population information can be accurately recorded.
The following detailed description is made with reference to the accompanying drawings and examples.
Description of the drawings:
FIG. 1 is a schematic block diagram;
FIG. 2 is a process flow for acquiring images;
fig. 3 is a trajectory prediction flow.
The specific embodiment is as follows:
with reference to fig. 1-3.
Step one, a monitoring method: monitoring a bird group by adopting high-frame-rate sequence images, identifying the types of bird targets, simultaneously carrying out real-time mark matching on bird target positions in continuous frames, recording bird flight tracks, and counting the number of birds according to mark numbers;
step two, the processing flow of the acquired image sequence is as follows: the observation sites of bird migration are selected from the places with a large number of waiting birds passing through in the migration period, such as sea islands far away from continents and mountain choke; obtaining continuous frame sequence images by adopting a high frame rate image sensor; the image acquisition process flow comprises the following steps: the method comprises the steps that a large target image is obtained for bird images in a threshold range of available discrimination types, a bird group image is obtained for images with the largest number of detected birds after filtering, and a frame sequence is obtained for all frame images from a first frame image to a last frame image which contain bird targets in sequence time arrangement;
step three, connected domain judgment:
judging the moving direction of the target, recording the mass center position of the target in the current frame, judging the moving range of the next frame according to the flying speed and the flying direction of birds, and searching the target in the moving range; according to the method, the connected domains where the same flying bird targets are located are matched according to the time sequence, so that the moving track of the flying bird can be marked, the newly-appearing targets in the field of view are marked, the shielding problem of the flying bird in different images can be effectively identified, and the number of flying bird groups can be counted accurately;
matching the continuous frame connected domain of the flying bird:
(1) After the current frame is subjected to target segmentation, marking a target position, numbering a target connected domain, 1, … …, i, and recording the centroid position of the target connected domain;
(2) Judging the movement direction of the flying bird according to the continuous frame movement coordinates of the target, and predicting the movement direction and the distance of the target of the next frame;
(3) After the next frame of image is segmented, carrying out connected domain centroid position marking, screening and matching the connected domain according to the predicted direction in the step (2), carrying out Euclidean distance calculation with the screened connected domain centroid, and carrying out connected domain screening according to the predicted distance range in the step (2);
(4) After the step (3) is finished, if the number n of the connected domains matched by the frame and the previous frame is 1, the matching is considered to be correct, and the existing connected domain numbers 1, … … and i are not updated; if the number n of the connected domains matched with the previous frame is an integer greater than 1, judging that the previous frame has a shielding area, and sequentially updating the matched connected domain numbers to i, i+1, … … and i+n-1;
(5) Sequentially numbering the newly added connected domains of each frame to ensure that marked birds in the video sequence images are not repeatedly counted;
fourthly, determining the mass center of the connected domain and predicting the track:
(1) Connected domain centroid determination
Let the image function be f (x, y), then the p+q standard moment of the image is:
wherein p and q are positive integers;
the target centroid position is:
wherein: x, y are image coordinates; m and N respectively represent the height and width of the image;
representing the centroid coordinates of the image;
(2) Track prediction:
calculating a target centroid by adopting a geometric moment, calculating a target distance of an adjacent frame by adopting an Euclidean distance, and carrying out target range prediction of a next frame according to the current frame, the ith frame, the first two frames, the ith-1 frame and the ith-2 frame and cached data;
the frame sequence prediction model and the method are as follows: according to the i-2 th frame centroid coordinates
With the i-1 frame target centroidCoordinates->
Predicting the possible centroid point position of the ith frame object +.>
Setting the point as a predicted point, and calculating the deviation angle of the track and the horizontal position +.>
Coordinates of centroid with object in frame i +.>
Calculating and predicting target position +.>
The prediction formula is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
in order to predict the deflection angle range in the horizontal direction, d is the predicted distance range, θ j D is the angle margin j Is the length allowance. .
Claims (1)
1. A serial image population identification method for monitoring migration of a waiting bird is characterized by comprising the following steps:
step one, a monitoring method: monitoring a bird group by adopting high-frame-rate sequence images, identifying the types of bird targets, simultaneously carrying out real-time mark matching on bird target positions in continuous frames, recording bird flight tracks, and counting the number of birds according to mark numbers;
step two, the processing flow of the acquired image sequence is as follows: the observation sites of bird migration are selected from the places with a large number of waiting birds passing through in the migration period, such as sea islands far away from continents and mountain choke; obtaining continuous frame sequence images by adopting a high frame rate image sensor; the image acquisition process flow comprises the following steps: the method comprises the steps that a large target image is obtained for bird images in a threshold range of available discrimination types, a bird group image is obtained for images with the largest number of detected birds after filtering, and a frame sequence is obtained for all frame images from a first frame image to a last frame image which contain bird targets in sequence time arrangement;
step three, connected domain judgment:
judging the moving direction of the target, recording the mass center position of the target in the current frame, judging the moving range of the next frame according to the flying speed and the flying direction of birds, and searching the target in the moving range; according to the method, the connected domains where the same flying bird targets are located are matched according to the time sequence, so that the moving track of the flying bird can be marked, the newly-appearing targets in the field of view are marked, the shielding problem of the flying bird in different images can be effectively identified, and the number of flying bird groups can be counted accurately;
matching the continuous frame connected domain of the flying bird:
(1) After completing target segmentation for the current frame, marking the target position, numbering the target connected domain by 1, … …, i, and recording the centroid position;
(2) Judging the movement direction of the flying bird according to the continuous frame movement coordinates of the target, and predicting the movement direction and the distance of the target of the next frame;
(3) After the next frame of image is segmented, carrying out connected domain centroid position marking, screening and matching the connected domain according to the predicted direction in the step (2), carrying out Euclidean distance calculation with the screened connected domain centroid, and carrying out connected domain screening according to the predicted distance range in the step (2);
(4) After the step (3) is finished, if the number n of the connected domains matched by the frame and the previous frame is 1, the matching is considered to be correct, and the existing connected domain numbers 1, … … and i are not updated; if the number n of the connected domains matched with the previous frame is an integer greater than 1, judging that the previous frame has a shielding area, and sequentially updating the matched connected domain numbers to i, i+1, … … and i+n-1;
(5) Sequentially numbering the newly added connected domains of each frame to ensure that marked birds in the video sequence images are not repeatedly counted;
fourthly, determining the mass center of the connected domain and predicting the track:
(1) Connected domain centroid determination
Let the image function be f (x, y), then the p+q standard moment of the image is:
wherein p and q are positive integers;
the target centroid position is:
wherein: x, y are image coordinates; m and N respectively represent the height and width of the image;
representing the centroid coordinates of the image;
(2) Track prediction:
calculating a target centroid by adopting a geometric moment, calculating a target distance of an adjacent frame by adopting an Euclidean distance, and carrying out target range prediction of a next frame according to the current frame, the ith frame, the first two frames, the ith-1 frame and the ith-2 frame and cached data;
the frame sequence prediction model and the method are as follows: according to the i-2 th frame centroid coordinates
And the i-1 frame target centroid coordinates
Predicting the possible centroid point position of the ith frame object +.>
Setting the point as a predicted point, and calculating the deviation angle of the track and the horizontal position +.>
Coordinates of centroid with object in frame i +.>
Calculating and predicting target position +.>
The prediction formula is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
in order to predict the deflection angle range in the horizontal direction, d is the predicted distance range, θ j D is the angle margin j Is the length allowance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011553426.1A CN112396626B (en) | 2020-12-24 | 2020-12-24 | Sequential image population identification method for monitoring migration of waiting birds |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011553426.1A CN112396626B (en) | 2020-12-24 | 2020-12-24 | Sequential image population identification method for monitoring migration of waiting birds |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112396626A CN112396626A (en) | 2021-02-23 |
| CN112396626B true CN112396626B (en) | 2023-06-27 |
Family
ID=74625520
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011553426.1A Active CN112396626B (en) | 2020-12-24 | 2020-12-24 | Sequential image population identification method for monitoring migration of waiting birds |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112396626B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113468439A (en) * | 2021-06-29 | 2021-10-01 | 北京市商汤科技开发有限公司 | Data processing method, device, equipment, storage medium and program product |
| CN113486940B (en) * | 2021-06-30 | 2023-10-31 | 浙江大学 | Adaptive bandwidth-based bird waiting migration track reconstruction method |
| CN115984309B (en) * | 2021-12-10 | 2024-03-15 | 北京百度网讯科技有限公司 | Method and apparatus for training image segmentation model and image segmentation |
| CN117611885A (en) * | 2023-11-17 | 2024-02-27 | 贵州省生物研究所 | Waiting bird ecological regulation and control method based on Canny edge detection |
| CN117591794B (en) * | 2024-01-18 | 2024-03-22 | 吉林省林业科学研究院(吉林省林业生物防治中心站) | Bird migration track prediction method based on time sequence |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2008092393A1 (en) * | 2007-01-25 | 2008-08-07 | Shanghai Yao Wei Industry Co, Ltd | Method of moving target tracking and number accounting |
| CN107480591A (en) * | 2017-07-10 | 2017-12-15 | 北京航空航天大学 | Flying bird detection method and device |
| CN109636771A (en) * | 2018-10-23 | 2019-04-16 | 中国船舶重工集团公司第七0九研究所 | Airbound target detection method and system based on image procossing |
| CN111968159A (en) * | 2020-08-28 | 2020-11-20 | 厦门大学 | Simple and universal fish video image track tracking method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10733739B2 (en) * | 2017-09-29 | 2020-08-04 | Nanjing Avatarmind Robot Technology Co., Ltd. | Method and system for displaying target image based on robot |
-
2020
- 2020-12-24 CN CN202011553426.1A patent/CN112396626B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2008092393A1 (en) * | 2007-01-25 | 2008-08-07 | Shanghai Yao Wei Industry Co, Ltd | Method of moving target tracking and number accounting |
| CN107480591A (en) * | 2017-07-10 | 2017-12-15 | 北京航空航天大学 | Flying bird detection method and device |
| CN109636771A (en) * | 2018-10-23 | 2019-04-16 | 中国船舶重工集团公司第七0九研究所 | Airbound target detection method and system based on image procossing |
| CN111968159A (en) * | 2020-08-28 | 2020-11-20 | 厦门大学 | Simple and universal fish video image track tracking method |
Non-Patent Citations (2)
| Title |
|---|
| "Statistical analysis of trajectories on Riemannian manifolds Bird migration, hurricane tracking and video surveillance";Jingyong Su等;《The Annals of Applied Statistics》;20141231;第8卷(第1期);第530-552页 * |
| "一种飞行目标跟踪方法";程科等;《2004年全国光电技术学术交流会论文集（下）》;20041231;第855-857页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112396626A (en) | 2021-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112396626B (en) | Sequential image population identification method for monitoring migration of waiting birds | |
| CN108986064B (en) | People flow statistical method, equipment and system | |
| CN106096577B (en) | A kind of target tracking method in camera distribution map | |
| CN113034548B (en) | Multi-target tracking method and system suitable for embedded terminal | |
| CN104200657B (en) | A kind of traffic flow parameter acquisition method based on video and sensor | |
| CN110852283A (en) | Helmet wearing detection and tracking method based on improved YOLOv3 | |
| CN106446002A (en) | Moving target-based video retrieval method for track in map | |
| CN104637058B (en) | A kind of volume of the flow of passengers identify statistical methods based on image information | |
| CN113012150A (en) | Feature-fused high-density rice field unmanned aerial vehicle image rice ear counting method | |
| CN111709372A (en) | Bird repelling method and device, computer equipment and storage medium | |
| Ling et al. | Simultaneous measurements of three-dimensional trajectories and wingbeat frequencies of birds in the field | |
| CN109239702B (en) | Airport low-altitude flying bird number statistical method based on target state set | |
| CN113759337B (en) | Three-dimensional ground penetrating radar real-time interpretation method and system for underground space data | |
| Corcoran et al. | Compound-V formations in shorebird flocks | |
| CN111640135A (en) | TOF camera pedestrian counting method based on hardware front end | |
| Gomaa et al. | Real-time algorithm for simultaneous vehicle detection and tracking in aerial view videos | |
| CN111325048A (en) | Personnel gathering detection method and device | |
| CN108986143A (en) | Target detection tracking method in a kind of video | |
| Shimasaki et al. | Hfr-video-based honeybee activity sensing | |
| CN114648557A (en) | Multi-target cooperative tracking method based on high-altitude visual angle and ground visual angle | |
| Wang et al. | Video SAR ground moving target indication based on multi-target tracking neural network | |
| CN106412507A (en) | Intelligent monitoring method and system of personnel flow | |
| CN110751077B (en) | Optical remote sensing picture ship detection method based on component matching and distance constraint | |
| Notz et al. | Extraction and assessment of naturalistic human driving trajectories from infrastructure camera and radar sensors | |
| Wang et al. | Low-slow-small target tracking using relocalization module |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |