CN113160071B - Satellite image automatic geometric correction method, system, medium and terminal equipment - Google Patents
Satellite image automatic geometric correction method, system, medium and terminal equipment Download PDFInfo
- Publication number
- CN113160071B CN113160071B CN202110266590.2A CN202110266590A CN113160071B CN 113160071 B CN113160071 B CN 113160071B CN 202110266590 A CN202110266590 A CN 202110266590A CN 113160071 B CN113160071 B CN 113160071B
- Authority
- CN
- China
- Prior art keywords
- image
- target satellite
- modis
- correction
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012937 correction Methods 0.000 title claims abstract description 163
- 238000000034 method Methods 0.000 title claims abstract description 78
- 238000012216 screening Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 19
- 238000011156 evaluation Methods 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 17
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 4
- 238000003702 image correction Methods 0.000 abstract description 6
- 230000007547 defect Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000002310 reflectometry Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an automatic geometric correction method, device, storage medium and terminal equipment for a domestic polar small satellite image, which are used for automatically and preferentially screening homologous sensor reference images for automatic geometric correction of a target satellite image; the whole scene image and the image are divided into four parts and nine parts for image enhancement processing, and then homonymous point pairs of the target satellite image and the reference image are extracted to obtain a geographic coordinate file for geometric correction; and carrying out correction precision evaluation on the images corrected by different methods of the target satellite so as to screen out an optimal correction scheme, and correcting the images of the target satellite according to the optimal correction scheme. The method can realize automatic optimal screening of the registration reference images of the homologous sensors, increase the selection of homonymous points by highlighting the detailed characteristics of the polar images through an image enhancement method, and screen an optimal geometric correction scheme according to the image correction precision so as to overcome the defect of low geometric positioning precision of polar small satellites in China.
Description
Technical Field
The invention relates to the technical field of remote sensing image processing, in particular to an automatic geometric correction method, an automatic geometric correction system, a storage medium and terminal equipment for satellite images.
Background
The existing image registration aiming at the homologous sensor is mainly carried out by adopting a characteristic-based method, but before massive data registration, the registration reference images and the homonym points of the homologous sensor are selected and selected in an automatic and optimal mode. In particular, the polar region image has the advantages that the ice and snow reflectivity is high, the textures of the ice cover surface are rare, and the feature points extracted on the ice cover surface by the scale-invariant feature transformation algorithm are fewer, so that the geometric correction of the polar region image is difficult.
Disclosure of Invention
The technical problem to be solved by the embodiment of the invention is to provide an automatic geometric correction method, an automatic geometric correction system, a storage medium and terminal equipment for satellite images, which can realize automatic optimal screening of registration reference images of homologous sensors, increase selection of homonymous points by highlighting detailed features of polar images through an image enhancement method, and screen out an optimal geometric correction scheme according to image correction precision so as to overcome the defect of low geometric positioning precision of polar satellites in China.
In order to solve the above technical problems, an embodiment of the present invention provides a satellite image automation geometric correction method applied to a polar small satellite, the method includes:
the reference images used for the automatic geometric correction of the target satellite images are preferentially screened from three aspects of far and near image acquisition time, space coverage degree and homonymous point pair numbers so as to obtain the optimal reference images used for the geometric correction of the target satellite images;
extracting homonymy points of the target satellite image and the MODIS image by adopting a preset scheme to obtain geographic coordinate files of each scheme for geometric correction; the method comprises the steps of extracting homonymous points through a whole scene image, performing image enhancement processing on the image by four parts and nine parts, and extracting homonymous points;
and carrying out correction precision evaluation on the image corrected by the target satellite image by adopting a preset scheme so as to screen out an optimal correction scheme, and correcting the target satellite image according to the optimal correction scheme.
Further, the method comprises the steps of preferentially screening the MODIS image for the automatic geometric correction of the target satellite image according to three aspects of far-near image acquisition time, spatial coverage degree and homonymous point pair number to obtain an optimal reference image for the geometric correction of the target satellite image, wherein the optimal reference image comprises the following specific steps:
obtaining the cloud MODIS images of the target satellite images on the same day in batches, and creating a first data index table of the target satellite images and the MODIS images;
preprocessing a target satellite image and an MODIS image, and obtaining the MODIS image consistent with the geographic range of the target satellite image by adopting consistent antarctic projection and 250m resolution;
preliminary screening is carried out on the MODIS image, the effective data range of the MODIS image covers more than 80% of the target image, and a second data index table of the target satellite image and the MODIS image is created;
and carrying out secondary screening on the MODIS images, extracting homonymous points of the target satellite and the MODIS image pairs according to the second data index table by a feature matching method, sorting and preferentially selecting corresponding MODIS images according to the number of the homonymous point pairs, and taking the MODIS image close to the shooting time of the target satellite image as a final registration reference image if the multiple-view MODIS data still have the same homonymous point pairs, and obtaining a third data index table for geometric correction.
Further, the homonymous points of the target satellite image and the MODIS image are extracted by adopting a preset scheme to obtain a geographic coordinate file for geometric correction of each scheme, specifically:
scheme one: after the target satellite whole-scene image and the corresponding MODIS whole-scene image are extracted through a scale invariant feature transform algorithm, sorting the same-name point pairs according to Euclidean distances of the same-name point pairs, removing points with the largest Euclidean distance of 10%, and outputting real geographic coordinate files of the rest of the same-name point pairs as a first corrected geographic coordinate file for correcting the target satellite image;
scheme II: dividing a target image rule into four parts, cutting corresponding MODIS images of 40 pixels in an extension direction, carrying out piecewise linear stretching enhancement processing on each image pair, extracting point pairs with the same name and the largest 30% of Euclidean distance through a scale invariant feature transform algorithm, outputting real geographic coordinate files of the rest points with the same name, merging the coordinate files and the first corrected geographic coordinate files, and removing repeated point pairs as second corrected geographic coordinate files for correcting the target satellite images;
scheme III: and dividing the target image rule into nine parts, obtaining real geographic coordinate files of the rest homonymous points after removing the point with the maximum Euclidean distance by using the second scheme, combining the coordinate file with the first corrected geographic coordinate file and the second corrected geographic coordinate file, and removing repeated point pairs to be used as a third corrected geographic coordinate file for correcting the target satellite image.
Further, the correction accuracy evaluation is performed on the images corrected by different methods on the target satellite image, so as to screen out an optimal correction scheme, and the correction is performed on the target satellite image according to the optimal correction scheme, specifically:
the three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of the target image by a quadratic polynomial method, so that the target satellite images corrected by different schemes are obtained;
extracting homonymous points from the three corrected target satellite images and the corresponding MODIS images again by using a scale invariant feature transform algorithm, calculating Euclidean distances between point pairs, evaluating the correction precision of geometric correction of different schemes, and screening out an optimal correction scheme;
and performing geometric correction on the target satellite image by using an optimal correction scheme.
In order to solve the above technical problem, an embodiment of the present invention further provides a satellite image automation geometry correction system, including:
the screening module is used for preferentially screening the reference image for the automatic geometric correction of the target satellite image from three aspects of image acquisition time far and near, space coverage degree and homonymous point pair number so as to obtain an optimal reference image for the geometric correction of the target satellite image;
the extraction module is used for extracting homonymous points of the target satellite image and the MODIS image by adopting a preset scheme so as to obtain geographic coordinate files for geometric correction of each scheme; the method comprises the steps of extracting homonymous points through a whole scene image, performing image enhancement processing on the image by four parts and nine parts, and extracting homonymous points;
the correction module is used for carrying out correction precision evaluation on the image corrected by the preset scheme on the target satellite image so as to screen out an optimal correction scheme and correcting the target satellite image according to the optimal correction scheme.
Further, the screening module is specifically used for,
obtaining the cloud MODIS images of the target satellite images on the same day in batches, and creating a first data index table of the target satellite images and the MODIS images;
preprocessing a target satellite image and an MODIS image, and obtaining the MODIS image consistent with the geographic range of the target satellite image by adopting consistent antarctic projection and 250m resolution;
preliminary screening is carried out on the MODIS image, the effective data range of the MODIS image covers more than 80% of the target image, and a second data index table of the target satellite image and the MODIS image is created;
and carrying out secondary screening on the MODIS images, extracting homonymous points of the target satellite and the MODIS image pairs according to the second data index table by a feature matching method, sorting and preferentially selecting corresponding MODIS images according to the number of the homonymous point pairs, and taking the MODIS image close to the shooting time of the target satellite image as a final registration reference image if the multiple-view MODIS data still have the same homonymous point pairs, and obtaining a third data index table for geometric correction.
Further, the extraction module is specifically used for,
scheme one: after the target satellite whole-scene image and the corresponding MODIS whole-scene image are extracted through a scale invariant feature transform algorithm, sorting the same-name point pairs according to Euclidean distances of the same-name point pairs, removing points with the largest Euclidean distance of 10%, and outputting real geographic coordinate files of the rest of the same-name point pairs as a first corrected geographic coordinate file for correcting the target satellite image;
scheme II: dividing a target image rule into four parts, cutting corresponding MODIS images of 40 pixels in an extension direction, carrying out piecewise linear stretching enhancement processing on each image pair, extracting point pairs with the same name and the largest 30% of Euclidean distance through a scale invariant feature transform algorithm, outputting real geographic coordinate files of the rest points with the same name, merging the coordinate files and the first corrected geographic coordinate files, and removing repeated point pairs as second corrected geographic coordinate files for correcting the target satellite images;
scheme III: and dividing the target image rule into nine parts, obtaining real geographic coordinate files of the rest homonymous points after removing the point with the maximum Euclidean distance by using the second scheme, combining the coordinate file with the first corrected geographic coordinate file and the second corrected geographic coordinate file, and removing repeated point pairs to be used as a third corrected geographic coordinate file for correcting the target satellite image.
Further, the correction module is specifically configured to,
the three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of the target image by a quadratic polynomial method, so that the target satellite images corrected by different schemes are obtained;
extracting homonymous points from the three corrected target satellite images and the corresponding MODIS images again by using a scale invariant feature transform algorithm, calculating Euclidean distances between point pairs, evaluating the correction precision of geometric correction of different schemes, and screening out an optimal correction scheme;
and performing geometric correction on the target satellite image by using an optimal correction scheme.
The embodiment of the invention also provides a computer readable storage medium, which comprises a stored computer program; the computer program controls the equipment of the computer readable storage medium to execute the satellite image automatic geometric correction method when running.
The embodiment of the invention also provides a terminal device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor realizes the satellite image automatic geometric correction method when executing the computer program.
Compared with the prior art, the embodiment of the invention provides a satellite image automation geometric correction method, which comprises the steps of firstly, preferentially screening a reference MODIS image for target satellite image automation geometric correction from three aspects of far and near image acquisition time, space coverage degree and homonymous point pair number so as to obtain an optimal reference image for target satellite image geometric correction; then, extracting homonymous points of the target satellite image and the MODIS image by adopting different methods, wherein the homonymous points are extracted through the whole scene image, the image is divided into four parts and nine parts for image enhancement processing, and then the homonymous points are extracted, so that geographic coordinate files for geometric correction of each scheme are obtained; and finally, carrying out correction precision evaluation on the images corrected by different methods on the target satellite images so as to screen out an optimal correction scheme, and correcting the target satellite images according to the optimal correction scheme. Compared with the prior art, the method can realize automatic optimal screening of the registration reference images of the homologous sensors, increase the selection of homonymous points by highlighting the detailed characteristics of the polar images through an image enhancement method, and screen an optimal geometric correction scheme according to the image correction precision so as to overcome the defect of low geometric positioning precision of the conventional polar small satellite.
Drawings
FIG. 1 is a flow chart of an automated geometric correction method for satellite images according to the present invention;
FIG. 2 is a data flow diagram of an automated geometric correction method for satellite images according to the present invention;
FIG. 3 is a diagram showing the effects of different image enhancement in an automatic geometric correction method for satellite images according to the present invention;
FIG. 4 is a schematic diagram showing an automated geometric correction method for satellite images according to the present invention;
FIG. 5 is a block diagram of an automated geometric correction method for satellite images according to the present invention;
fig. 6 is a block diagram of a terminal device according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, are intended to fall within the scope of the present invention.
It should be noted that, the step numbers herein are only for convenience of explanation of the specific embodiments, and are not used as limiting the order of execution of the steps. The method provided in this embodiment may be executed by a relevant server, and the following description will take the server as an execution body as an example.
As shown in fig. 1 to 4, the method for automatically correcting geometry of satellite images provided by the embodiment of the invention is applied to a polar small satellite, and the method includes steps S11 to S14:
step S11, the reference image for the automatic geometric correction of the target satellite image is preferentially screened from three aspects of image acquisition time far and near, space coverage degree and homonymous point pair number so as to obtain the optimal reference image for the geometric correction of the target satellite image.
Specifically, obtaining MODIS images with less cloud cover of a target satellite image on the same day in batches, and creating a first data index table of the target satellite image and the MODIS images; preprocessing the target satellite image and the MODIS image, and obtaining the MODIS image consistent with the geographic range of the target satellite image through image clipping by adopting consistent projection (south pole projection) and resolution (250 m); preliminary screening is carried out on the MODIS image, images, the effective data range of which covers more than a certain target value of the target satellite image, are screened, and a second data index table of the target satellite image and the MODIS image is created; and carrying out secondary screening on the MODIS images, extracting homonymous points of each target satellite and the MODIS image pair according to the second data index table by a feature matching method, sorting and preferentially selecting corresponding MODIS images according to the number of the homonymous point pairs, and taking the MODIS image close to the shooting time of the target satellite as a final registration reference image and obtaining a third data index table for geometric correction if the multiple-view MODIS data have the same homonymous point pairs.
Further, taking the polar small satellite "Beijing operator No. (development code: BNU-1) satellite of China as an example, the MODIS images (MODIS reflectivity data of the United states) of the same day corresponding to the images of the BNU-1 satellite are downloaded in batches, and a first data index table of the polar small satellite and the BNU-1 satellite is established. BNU-1 data defines the Antarctic projection (set resolution to 250 m) (same as MODIS data); the MODIS image extracts the radiance and recovers the geographic coordinates of a single wave band to output GeoTIFF, defines the projection of a south pole (the resolution is set to be 250 m), and cuts the MODIS image (the BNU-1 image with the resolution of 250m extends 40 pixels (namely 10 km) to cut the corresponding MODIS image). When the range of the MODIS effective data after cutting/the range of the corresponding BNU-1 data is more than or equal to 0.8, a second data index table of the BNU-1 satellite image and the MODIS image is created. And extracting homonymous point pairs of the BNU-1 satellite image and the corresponding MODIS image by using a Scale Invariant Feature Transform (SIFT) matching method according to the second data index table. And selecting the corresponding MODIS images preferentially according to the number sequence of the homonymous point pairs, and if the multi-view MODIS data still have the homonymous point pairs with the same quantity, taking the MODIS data close to the BNU-1 shooting time as a final registration reference image to obtain a third data index table finally used for registration.
Step S12, extracting homonymous points of the target satellite image and the MODIS image by adopting a preset scheme to obtain a geographic coordinate file of each scheme for geometric correction; the method comprises the steps of extracting homonymous points through a whole scene image, performing image enhancement processing on the image by four parts and nine parts, and extracting homonymous points.
Wherein, scheme one: extracting the homonymy point coordinates of the homonymy point pair by a whole point image method to obtain a corrected geographic coordinate file, wherein the method specifically comprises the following steps: after extracting homonymous point pairs from the target satellite whole-scene image and the corresponding cut MODIS whole-scene image through a SIFT algorithm, sorting according to Euclidean distances of the homonymous point pairs, removing points with the largest Euclidean distance of 10%, and outputting real geographic coordinate files of the rest homonymous point pairs as a first corrected geographic coordinate file for correcting the automatic geometric correction of the target satellite image.
Specifically, the BNU-1 whole-scene image and the corresponding cut MODIS whole-scene image (according to a third data index table) are extracted into homonymous point pairs by using a SIFT algorithm, the euclidean distances of the homonymous point pairs are sorted, 10% of the points with the largest euclidean distances are removed, and the real geographic coordinate files of the rest homonymous point pairs are output to serve as first corrected geographic coordinate files for correcting the BNU-1 satellite image.
Wherein, scheme II: dividing the target satellite image rule into four parts (2 x 2 = 4), cutting corresponding MODIS images from each part to 40 pixels, carrying out piecewise linear stretching enhancement processing on each image pair, extracting the homonym point pair through a SIFT algorithm, removing the point with the largest Euclidean distance, outputting the real geographic coordinate file of the rest homonym points, merging the coordinate file and the first corrected geographic coordinate file, and removing the repeated point pair to be used as a second corrected geographic coordinate file for correcting the BNU-1 satellite image.
Specifically, referring to fig. 3, the BNU-1 image rule is clipped into 4 parts (2×2=4), and the four parts are respectively extended with 40 pixels of clipping corresponding MODIS data. And respectively carrying out image piecewise linear stretching enhancement treatment on the four groups of image pairs, and then using a SIFT algorithm to extract the same name point pairs. The comparison of the homonymous point pairs extracted by the different stretching methods in fig. 3 is: (a) no stretching; (b) linear stretching; (c) piecewise linear stretching; (d) gaussian stretching; (e) histogram equalization stretching; (f) Root mean square stretching (comparing the results of six stretching modes including no stretching, linearity, piecewise linearity, gaussian, histogram equalization and root mean square, finding out more homonymous points found by the piecewise linear stretching mode), eliminating the point with the largest Euclidean distance by 30%, and outputting the true geographic coordinate files of the rest homonymous points. And combining the coordinate file with the first corrected geographic coordinate file, and then removing the repeated point pairs to be used as a second corrected geographic coordinate file for correcting the BNU-1 satellite image. It can be understood that the feature of ice and snow in the polar region can be highlighted by acquiring the homonymy points after image enhancement, so that more homonymy points can be found.
Wherein, scheme III: dividing the target satellite image and the MODIS image into nine parts (3*3 =9, extracting the same-name point pairs, removing the point with the largest Euclidean distance of 30%, and outputting the real geographic coordinate files of the rest same-name points (the same method and the second scheme), combining the coordinate files with the first corrected geographic coordinate file and the second corrected geographic coordinate file, and removing the repeated point pairs as a third corrected geographic coordinate file for correcting the target satellite image.
Specifically, BNU-1 satellite images and MODIS images are uniformly divided into 9 parts for piecewise linear enhancement, then homonymous point pairs are extracted, 30% of the homonymous point pairs are removed, and then coordinate files are output (the method is the same as scheme II). And after the first corrected geographic coordinate file and the second corrected geographic coordinate file are combined, eliminating the repeated point pairs to be used as a third corrected geographic coordinate file for correcting BNU-1 satellite images.
And S13, performing correction accuracy evaluation on the images corrected by different schemes on the target satellite image to screen out an optimal correction scheme, and correcting the target satellite image according to the optimal correction scheme.
Specifically, three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of the target satellite image by a quadratic polynomial method to obtain target satellite images corrected by different schemes; extracting homonymous points from the corrected BNU-1 satellite images (three) and the corresponding MODIS images again by using a SIFT algorithm, calculating Euclidean distance between point pairs, evaluating the correction precision after geometric correction of different schemes, and screening out an optimal correction scheme; and performing geometric correction on the BNU-1 satellite image by using an optimal correction scheme.
Further, the three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of BNU-1 satellite images by a quadratic polynomial method to obtain images corrected by different schemes; and extracting homonymous points from the corrected BNU-1 satellite images (three) and the corresponding MODIS images again by using a SIFT algorithm, calculating Euclidean distance between point pairs, evaluating the correction precision (table 1) after geometric correction of different schemes, and screening out the optimal correction scheme.
Table 1 three schemes for estimating the number of homonymous points and correction accuracy of automatic geometric correction of BNU-1 satellite images
(taking BNU-1 satellite as an example, 12/18/09/04/51)
The automatic geometric correction technology suitable for the polar small satellite image is applied to the BNU-1 satellite image for geometric correction, corresponding MODIS data are respectively overlapped on the images before and after correction, and the effects before and after correction are compared (taking the image shot by BNU-1 satellite 2019, 12 months, 18 days, 09:04:51 as an example), and the result is shown in fig. 4.
In fig. 4, fig. 4a shows a BNU-1 0 level image and fig. 4b shows an image corrected by using an automatic geometric correction technique suitable for a domestic polar small satellite image. Fig. 4c, e, g are enlarged detail of fig. 4a, respectively, sea ice boundary, ice cover, and cloudy portion. Fig. 4d, f, h are enlarged views of the corresponding positions of the corrected images using the technique of the present invention. From the result, the automatic geometric correction technology suitable for the domestic polar region small satellite image has obvious advantages in the following aspects, firstly, the technology can automatically select and screen the registration reference image of the homologous sensor (can select the optimal scene from tens of scenes of data every day), and the manual intervention brought by the aspects of image data range, shooting time, data quality and the like is also needed due to the large satellite data volume. Secondly, the technology carries out partial enhancement processing on the image, highlights the detail characteristics of the polar region image, and enables the SIFT operator to extract homonymous point pairs more uniformly on the polar region image. Thirdly, the technology can achieve optimal correction precision according to the preferential selection of correction schemes of different images. Fourth, the geometric accuracy of the corrected images by the technology is obviously improved (as 6221.77m is improved to 243.48 m), a series of high-accuracy registered data products can be automatically produced, and effective support and guarantee are provided for improving and promoting the application of polar region small satellite data.
The embodiment of the invention provides a satellite image automatic geometric correction method, which comprises the steps of firstly, preferentially screening a reference MODIS image for target satellite image automatic geometric correction from three aspects of far-near image acquisition time, space coverage degree and homonymous point pair number so as to obtain an optimal reference image for target satellite image geometric correction; then, extracting homonymous points of the target satellite image and the MODIS image by adopting different methods, wherein the homonymous points are extracted through the whole scene image, the image is divided into four parts and nine parts for image enhancement processing, and then the homonymous points are extracted, so that geographic coordinate files for geometric correction of each scheme are obtained; and finally, carrying out correction precision evaluation on the images corrected by different methods on the target satellite images so as to screen out an optimal correction scheme, and correcting the target satellite images according to the optimal correction scheme. Compared with the prior art, the method can realize automatic optimal screening of the registration reference images of the homologous sensors, increase the selection of homonymous points by highlighting the detailed characteristics of the polar images through an image enhancement method, and screen an optimal geometric correction scheme according to the image correction precision, so as to overcome the defect of low geometric positioning precision of polar small satellites in China and meet the actual application demands.
Fig. 5 is a block diagram of an automated geometric correction system for satellite images according to the present invention, where the system includes:
the screening module 21 is configured to preferentially screen the reference image for the automatic geometric correction of the target satellite image from three aspects of image acquisition time, spatial coverage degree and number of homonymous points, so as to obtain an optimal reference image for the geometric correction of the target satellite image.
Further, the screening module 21 is specifically configured to,
obtaining the cloud MODIS images of the target satellite images on the same day in batches, and creating a first data index table of the target satellite images and the MODIS images;
preprocessing a target satellite image and an MODIS image, and obtaining the MODIS image consistent with the geographic range of the target satellite image by adopting consistent antarctic projection and 250m resolution;
preliminary screening is carried out on the MODIS image, the effective data range of the MODIS image covers more than 80% of the target image, and a second data index table of the target satellite image and the MODIS image is created;
and carrying out secondary screening on the MODIS images, extracting homonymous points of the target satellite and the MODIS image pairs according to the second data index table by a feature matching method, sorting and preferentially selecting corresponding MODIS images according to the number of the homonymous point pairs, and taking the MODIS image close to the shooting time of the target satellite image as a final registration reference image if the multiple-view MODIS data still have the same homonymous point pairs, and obtaining a third data index table for geometric correction.
The extracting module 22 is configured to extract homonymous points of the target satellite image and the MODIS image by using a preset scheme to obtain a geographic coordinate file for geometric correction of each scheme; the method comprises the steps of extracting homonymous points through a whole scene image, performing image enhancement processing on the image by four parts and nine parts, and extracting homonymous points.
Further, the extraction module 22 is specifically configured to,
scheme one: after the target satellite whole-scene image and the corresponding MODIS whole-scene image are extracted through a scale invariant feature transform algorithm, sorting the same-name point pairs according to Euclidean distances of the same-name point pairs, removing points with the largest Euclidean distance of 10%, and outputting real geographic coordinate files of the rest of the same-name point pairs as a first corrected geographic coordinate file for correcting the target satellite image;
scheme II: dividing a target image rule into four parts, cutting corresponding MODIS images of 40 pixels in an extension direction, carrying out piecewise linear stretching enhancement processing on each image pair, extracting point pairs with the same name and the largest 30% of Euclidean distance through a scale invariant feature transform algorithm, outputting real geographic coordinate files of the rest points with the same name, merging the coordinate files and the first corrected geographic coordinate files, and removing repeated point pairs as second corrected geographic coordinate files for correcting the target satellite images;
scheme III: and dividing the target image rule into nine parts, obtaining real geographic coordinate files of the rest homonymous points after removing the point with the maximum Euclidean distance by using the second scheme, combining the coordinate file with the first corrected geographic coordinate file and the second corrected geographic coordinate file, and removing repeated point pairs to be used as a third corrected geographic coordinate file for correcting the target satellite image.
The correction module 23 is configured to perform correction accuracy evaluation on an image corrected by using a preset scheme on a target satellite image, so as to screen an optimal correction scheme, and correct the target satellite image according to the optimal correction scheme.
Further, the correction module 23 is configured, in particular,
the three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of the target image by a quadratic polynomial method, so that the target satellite images corrected by different schemes are obtained;
extracting homonymous points from the three corrected target satellite images and the corresponding MODIS images again by using a scale invariant feature transform algorithm, calculating Euclidean distances between point pairs, evaluating the correction precision of geometric correction of different schemes, and screening out an optimal correction scheme;
and performing geometric correction on the target satellite image by using an optimal correction scheme.
According to the satellite image automatic geometric correction system provided by the embodiment of the invention, MODIS data of a target satellite on the same day as the running of the target satellite is obtained in batches, and the MODIS data is subjected to data preprocessing to obtain preprocessed MODIS data; sequentially performing primary screening and secondary screening on the preprocessed MODIS data to extract homonymous point pairs of the target satellite and the MODIS data; extracting homonymy point coordinates of the homonymy point pairs by using an integral point image and image segmentation enhancement method to obtain a corrected geographic coordinate file; and carrying out correction precision evaluation on the corrected geographic coordinate file to screen out an optimal correction scheme, and correcting the satellite image of the target satellite according to the optimal correction scheme. Compared with the prior art, the method can reduce the low positioning precision of the polar small satellite, realize automatic preferential screening of the registration reference images of the heterogeneous sensor, increase the selection of homonymous points based on the characteristics, and meet the actual application demands.
The embodiment of the invention also provides a computer readable storage medium, which comprises a stored computer program; the computer program controls the equipment where the computer readable storage medium is located to execute the satellite image correction method when running.
An embodiment of the present invention further provides a terminal device, referring to fig. 6, which is a block diagram of a preferred embodiment of a terminal device provided by the present invention, where the terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10, and the processor 10 implements the satellite image correction method when executing the computer program.
Preferably, the computer program may be partitioned into one or more modules/units (e.g., computer program 1, computer program 2, & gtthe & lt- & gt, & lt- & gt) that are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments are used for describing the execution of the computer program in the terminal device.
The processor 10 may be a central processing unit (Central Processing Unit, CPU), it may be a microprocessor, it may be other general purpose processor, it may be a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), an off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc., or it may be any conventional processor, the processor 10 being a control center of the terminal device, with various interfaces and lines connecting the various parts of the terminal device.
The memory 20 mainly includes a program storage area, which may store an operating system, application programs required for at least one function, and the like, and a data storage area, which may store related data and the like. In addition, the memory 20 may be a high-speed random access memory, a nonvolatile memory such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), etc., or the memory 20 may be other volatile solid-state memory devices.
It should be noted that the above-mentioned terminal device may include, but is not limited to, a processor, a memory, and those skilled in the art will understand that the structural block diagram of fig. 6 is merely an example of the terminal device, and does not constitute limitation of the terminal device, and may include more or less components than those illustrated, or may combine some components, or different components.
In summary, according to the method, the system, the storage medium and the terminal device for automatically correcting the geometric image of the satellite, which are provided by the embodiment of the invention, firstly, MODIS data of a target satellite on a current day of operation are obtained in batches, and data preprocessing is performed on the MODIS data to obtain preprocessed MODIS data; sequentially performing primary screening and secondary screening on the preprocessed MODIS data to extract homonymous point pairs of the target satellite and the MODIS data; extracting homonymy point coordinates of the homonymy point pairs by using an integral point image and image segmentation enhancement method to obtain a corrected geographic coordinate file; and carrying out correction precision evaluation on the corrected geographic coordinate file to screen out an optimal correction scheme, and correcting the satellite image of the target satellite according to the optimal correction scheme. Compared with the prior art, the method can reduce the low positioning precision of the polar small satellite, realize automatic preferential screening of the registration reference images of the heterogeneous sensor, increase the selection of homonymous points based on the characteristics, and meet the actual application demands.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.
Claims (6)
1. An automatic geometric correction method for satellite images, which is applied to polar satellites, is characterized in that the method comprises the following steps:
the MODIS image used for the automatic geometric correction of the target satellite image is preferentially screened from three aspects of far and near image acquisition time, space coverage degree and homonymous point pair number so as to obtain an optimal reference image used for the geometric correction of the target satellite image; the method comprises the following steps:
obtaining the cloud MODIS images of the target satellite images on the same day in batches, and creating a first data index table of the target satellite images and the MODIS images;
preprocessing a target satellite image and an MODIS image, and obtaining the MODIS image consistent with the geographic range of the target satellite image by adopting consistent antarctic projection and 250m resolution;
preliminary screening is carried out on the MODIS image, the effective data range of the MODIS image covers more than 80% of the target satellite image, and a second data index table of the target satellite image and the MODIS image is created;
performing secondary screening on the MODIS images, extracting homonymous points of target satellite and MODIS image pairs according to the second data index table by a feature matching method, sorting and preferentially selecting corresponding MODIS images according to the number of the homonymous point pairs, and taking the MODIS image close to the shooting time of the target satellite image as a final registration reference image if the same-name point pairs exist in the multi-view MODIS data, and obtaining a third data index table for geometric correction;
extracting homonymy points of the target satellite image and the MODIS image by adopting a preset scheme to obtain geographic coordinate files of each scheme for geometric correction; the method comprises the steps of extracting homonymous points through a whole scene image, performing image enhancement processing on the image by four parts and nine parts, and extracting homonymous points; the method comprises the following steps:
scheme one: extracting homonymy point pairs from the target satellite whole-scene image and the MODIS whole-scene image correspondingly cut according to the third data index table through a scale invariant feature transform algorithm, sorting according to Euclidean distances of the homonymy point pairs, removing points with the maximum Euclidean distance by 10%, and outputting real geographic coordinate files of the rest homonymy point pairs as a first corrected geographic coordinate file for correcting the target satellite image;
scheme II: dividing a target satellite image rule into four parts, cutting corresponding MODIS images from 40 pixels in an extension direction, carrying out piecewise linear stretching enhancement processing on each image pair, extracting homonym point pairs through a scale invariant feature transform algorithm, eliminating points with the maximum Euclidean distance, outputting real geographic coordinate files of other homonym points, merging the coordinate files and the first corrected geographic coordinate files, and eliminating repeated point pairs to serve as a second corrected geographic coordinate file for correcting the target satellite image;
scheme III: dividing the target satellite image rule into nine parts, obtaining real geographic coordinate files of other homonymous points after removing points with the maximum Euclidean distance of 30%, merging the coordinate files with the first corrected geographic coordinate file and the second corrected geographic coordinate file, and removing repeated point pairs as a third corrected geographic coordinate file for correcting the target satellite image;
the three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of the target satellite image by a quadratic polynomial method, so that the target satellite images corrected by different schemes are obtained;
and carrying out correction precision evaluation on the image corrected by the target satellite image by adopting a preset scheme so as to screen out an optimal correction scheme, and correcting the target satellite image according to the optimal correction scheme.
2. The method for automatically correcting geometry of satellite images according to claim 1, wherein the correction accuracy evaluation is performed on images corrected by different methods on the target satellite images to screen out an optimal correction scheme, and the correction is performed on the target satellite images according to the optimal correction scheme, specifically:
extracting homonymous points from the three corrected target satellite images and the corresponding MODIS images again by using a scale invariant feature transform algorithm, calculating Euclidean distances between point pairs, evaluating the correction precision of geometric correction of different schemes, and screening out an optimal correction scheme;
and performing geometric correction on the target satellite image by using an optimal correction scheme.
3. A satellite image automated geometry correction system, the system comprising:
the screening module is used for preferentially screening the MODIS image for the automatic geometric correction of the target satellite image from three aspects of far and near image acquisition time, space coverage degree and homonymous point pair number so as to obtain an optimal reference image for the geometric correction of the target satellite image; the method comprises the following steps:
obtaining the cloud MODIS images of the target satellite images on the same day in batches, and creating a first data index table of the target satellite images and the MODIS images;
preprocessing a target satellite image and an MODIS image, and obtaining the MODIS image consistent with the geographic range of the target satellite image by adopting consistent antarctic projection and 250m resolution;
preliminary screening is carried out on the MODIS image, the effective data range of the MODIS image covers more than 80% of the target satellite image, and a second data index table of the target satellite image and the MODIS image is created;
performing secondary screening on the MODIS images, extracting homonymous points of target satellite and MODIS image pairs according to the second data index table by a feature matching method, sorting and preferentially selecting corresponding MODIS images according to the number of the homonymous point pairs, and taking the MODIS image close to the shooting time of the target satellite image as a final registration reference image if the same-name point pairs exist in the multi-view MODIS data, and obtaining a third data index table for geometric correction;
the extraction module is used for extracting homonymous points of the target satellite image and the MODIS image by adopting a preset scheme so as to obtain geographic coordinate files for geometric correction of each scheme; the method comprises the steps of extracting homonymous points through a whole scene image, performing image enhancement processing on the image by four parts and nine parts, and extracting homonymous points; the method comprises the following steps:
scheme one: extracting homonymy point pairs from the target satellite whole-scene image and the MODIS whole-scene image correspondingly cut according to the third data index table through a scale invariant feature transform algorithm, sorting according to Euclidean distances of the homonymy point pairs, removing points with the maximum Euclidean distance by 10%, and outputting real geographic coordinate files of the rest homonymy point pairs as a first corrected geographic coordinate file for correcting the target satellite image;
scheme II: dividing a target satellite image rule into four parts, cutting corresponding MODIS images from 40 pixels in an extension direction, carrying out piecewise linear stretching enhancement processing on each image pair, extracting homonym point pairs through a scale invariant feature transform algorithm, eliminating points with the maximum Euclidean distance, outputting real geographic coordinate files of other homonym points, merging the coordinate files and the first corrected geographic coordinate files, and eliminating repeated point pairs to serve as a second corrected geographic coordinate file for correcting the target satellite image;
scheme III: dividing the target satellite image rule into nine parts, obtaining real geographic coordinate files of other homonymous points after removing points with the maximum Euclidean distance of 30%, merging the coordinate files with the first corrected geographic coordinate file and the second corrected geographic coordinate file, and removing repeated point pairs as a third corrected geographic coordinate file for correcting the target satellite image;
the three corrected geographic coordinate files obtained by the three schemes are respectively used for geometric correction of the target satellite image by a quadratic polynomial method, so that the target satellite images corrected by different schemes are obtained;
the correction module is used for carrying out correction precision evaluation on the image corrected by the preset scheme on the target satellite image so as to screen out an optimal correction scheme and correcting the target satellite image according to the optimal correction scheme.
4. The satellite image automated geometry correction system of claim 3, wherein the correction module is configured to,
extracting homonymous points from the three corrected target satellite images and the corresponding MODIS images again by using a scale invariant feature transform algorithm, calculating Euclidean distances between point pairs, evaluating the correction precision of geometric correction of different schemes, and screening out an optimal correction scheme;
and performing geometric correction on the target satellite image by using an optimal correction scheme.
5. A computer readable storage medium, wherein the computer readable storage medium comprises a stored computer program; wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the satellite image automated geometry correction method of any of claims 1-2.
6. A terminal device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the satellite image automated geometry correction method of any one of claims 1-2 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110266590.2A CN113160071B (en) | 2021-03-11 | 2021-03-11 | Satellite image automatic geometric correction method, system, medium and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110266590.2A CN113160071B (en) | 2021-03-11 | 2021-03-11 | Satellite image automatic geometric correction method, system, medium and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113160071A CN113160071A (en) | 2021-07-23 |
| CN113160071B true CN113160071B (en) | 2023-11-07 |
Family
ID=76886800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110266590.2A Active CN113160071B (en) | 2021-03-11 | 2021-03-11 | Satellite image automatic geometric correction method, system, medium and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113160071B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663028A (en) * | 2012-03-23 | 2012-09-12 | 北京师范大学 | Method suitable for fast spatially-indexing global digital elevation model and remote sensing image data |
| CN103034988A (en) * | 2012-12-18 | 2013-04-10 | 武汉大学 | Space-time quantitative remote sensing fusion method of arbitrary number of sensors |
| CN105046251A (en) * | 2015-08-04 | 2015-11-11 | 中国资源卫星应用中心 | Automatic ortho-rectification method based on remote-sensing image of environmental No.1 satellite |
| CN107657597A (en) * | 2017-10-19 | 2018-02-02 | 中国科学院遥感与数字地球研究所 | Cross-platform moon base earth observation image automatic geometric correction method |
| CN109903352A (en) * | 2018-12-24 | 2019-06-18 | 中国科学院遥感与数字地球研究所 | A kind of seamless orthography production method in the big region of satellite remote-sensing image |
| WO2019157348A1 (en) * | 2018-02-09 | 2019-08-15 | The Board Of Trustees Of The University Of Illinois | A system and method to fuse multiple sources of optical data to generate a high-resolution, frequent and cloud-/gap-free surface reflectance product |
| CN111354054A (en) * | 2020-03-13 | 2020-06-30 | 中山大学 | Polar region visible light remote sensing self-adaptive mapping method |
| CN111598772A (en) * | 2020-04-03 | 2020-08-28 | 国家卫星气象中心(国家空间天气监测预警中心) | Polar orbit meteorological satellite image display system and display method |
| CN112381864A (en) * | 2020-12-08 | 2021-02-19 | 兰州交通大学 | Multi-source multi-scale high-resolution remote sensing image automatic registration technology based on antipodal geometry |
| CN112419350A (en) * | 2020-11-20 | 2021-02-26 | 武汉大学 | Remote sensing image automatic geometric registration method and system based on ground object boundary information |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10802135B2 (en) * | 2016-12-21 | 2020-10-13 | The Boeing Company | Method and apparatus for raw sensor image enhancement through georegistration |
| WO2019062166A1 (en) * | 2017-09-30 | 2019-04-04 | 中国科学院遥感与数字地球研究所 | Method for automatic geometric correction of cross-platform moon-based earth observation imaging |
-
2021
- 2021-03-11 CN CN202110266590.2A patent/CN113160071B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663028A (en) * | 2012-03-23 | 2012-09-12 | 北京师范大学 | Method suitable for fast spatially-indexing global digital elevation model and remote sensing image data |
| CN103034988A (en) * | 2012-12-18 | 2013-04-10 | 武汉大学 | Space-time quantitative remote sensing fusion method of arbitrary number of sensors |
| CN105046251A (en) * | 2015-08-04 | 2015-11-11 | 中国资源卫星应用中心 | Automatic ortho-rectification method based on remote-sensing image of environmental No.1 satellite |
| CN107657597A (en) * | 2017-10-19 | 2018-02-02 | 中国科学院遥感与数字地球研究所 | Cross-platform moon base earth observation image automatic geometric correction method |
| WO2019157348A1 (en) * | 2018-02-09 | 2019-08-15 | The Board Of Trustees Of The University Of Illinois | A system and method to fuse multiple sources of optical data to generate a high-resolution, frequent and cloud-/gap-free surface reflectance product |
| CN109903352A (en) * | 2018-12-24 | 2019-06-18 | 中国科学院遥感与数字地球研究所 | A kind of seamless orthography production method in the big region of satellite remote-sensing image |
| CN111354054A (en) * | 2020-03-13 | 2020-06-30 | 中山大学 | Polar region visible light remote sensing self-adaptive mapping method |
| CN111598772A (en) * | 2020-04-03 | 2020-08-28 | 国家卫星气象中心(国家空间天气监测预警中心) | Polar orbit meteorological satellite image display system and display method |
| CN112419350A (en) * | 2020-11-20 | 2021-02-26 | 武汉大学 | Remote sensing image automatic geometric registration method and system based on ground object boundary information |
| CN112381864A (en) * | 2020-12-08 | 2021-02-19 | 兰州交通大学 | Multi-source multi-scale high-resolution remote sensing image automatic registration technology based on antipodal geometry |
Non-Patent Citations (3)
| Title |
|---|
| "京师一号"极地小卫星宽幅影像辐射校正方法研究;张卓宇 等;《北京师范大学学报(自然科学版)》;第59卷(第01期);104-112 * |
| Accuracy evaluation on geolocation of the Chinese first polar microsatellite (Ice pathfinder) imagery;Ying Zhang 等;《remote sensing》;第13卷(第4278期);1-20 * |
| Automated radiation and geometric correction processing method of polar-observing cubesat BNU-1 images;Y. Zhang 等;《AGU Fall Meeting 2020 abstracts》;A088-0004 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113160071A (en) | 2021-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9396583B2 (en) | Method of modelling buildings on the basis of a georeferenced image | |
| CN110781756A (en) | Urban road extraction method and device based on remote sensing image | |
| CN108573184B (en) | Two-dimensional code positioning method, module and computer readable storage medium | |
| CN111310710A (en) | Face detection method and system | |
| CN114119644B (en) | Template matching method based on edge features | |
| CN116170569A (en) | Hyperspectral image forward shooting correction method and device and storage medium | |
| CN113160071B (en) | Satellite image automatic geometric correction method, system, medium and terminal equipment | |
| CN109815763A (en) | Detection method, device and the storage medium of two dimensional code | |
| CN111062341B (en) | Video image area classification method, device, equipment and storage medium | |
| CN114092326A (en) | Web map tile updating processing method and device | |
| CN111476308B (en) | Remote sensing image classification method and device based on priori geometric constraint and electronic equipment | |
| CN117292281A (en) | Open-field vegetable detection method, device, equipment and medium based on unmanned aerial vehicle image | |
| CN114092695B (en) | ROI extraction method and device based on segmentation model | |
| CN111630569B (en) | Binocular matching method, visual imaging device and device with storage function | |
| CN114913105A (en) | Laser point cloud fusion method and device, server and computer readable storage medium | |
| CN112508065B (en) | Robot and positioning method and device thereof | |
| KR101775124B1 (en) | System and method for automatic satellite image processing for improvement of location accuracy | |
| CN114998630A (en) | Ground-to-air image registration method from coarse to fine | |
| CN114140581A (en) | Automatic modeling method and device, computer equipment and storage medium | |
| JPH08161474A (en) | Method for correcting registration between images by different kinds of sensors | |
| CN111581407A (en) | Method, device and medium for constructing global geographic positioning reference image database | |
| CN110827243A (en) | Method and device for detecting abnormity of coverage area of grid beam | |
| US20240257363A1 (en) | Alignment of dense point clouds of a physical object | |
| CN115660939B (en) | Method, system, equipment and storage medium for optical remote sensing data batch processing | |
| CN117201708B (en) | Unmanned aerial vehicle video stitching method, device, equipment and medium with position information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |