CN110992257A - Remote sensing image sensitive information automatic shielding method and device based on deep learning - Google Patents
Remote sensing image sensitive information automatic shielding method and device based on deep learning Download PDFInfo
- Publication number
- CN110992257A CN110992257A CN201911329520.6A CN201911329520A CN110992257A CN 110992257 A CN110992257 A CN 110992257A CN 201911329520 A CN201911329520 A CN 201911329520A CN 110992257 A CN110992257 A CN 110992257A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- sensing image
- sensitive information
- image
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013135 deep learning Methods 0.000 title claims abstract description 19
- 238000001514 detection method Methods 0.000 claims description 22
- 238000011176 pooling Methods 0.000 claims description 14
- 238000005070 sampling Methods 0.000 claims description 6
- 239000011800 void material Substances 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention discloses a remote sensing image sensitive information automatic shielding method and device based on deep learning. The method comprises the following steps: acquiring a remote sensing image to be processed; determining coordinate information of sensitive information contained in the remote sensing image to be processed, and converting pixel values of a region corresponding to the coordinate information in the remote sensing image to be processed into preset pixel values to obtain a candidate remote sensing image; inputting the candidate remote sensing image into a model obtained by pre-training to obtain a target remote sensing image obtained by pixel filling of a preset pixel value area in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of preset areas of the sample remote sensing images into preset pixel values according to the sample remote sensing images. By applying the scheme provided by the embodiment of the invention, sensitive information can be ensured not to be disclosed, and the image quality of the issued remote sensing image is not influenced.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a remote sensing image sensitive information automatic shielding method and device based on deep learning.
Background
At present, the resolution of remote sensing satellites is higher and higher, and sensitive information which does not meet public requirements, such as special buildings, fixed facilities and the like, may exist in images shot by the remote sensing satellites. For the remote sensing image containing the sensitive information, the remote sensing image can be released after being processed.
In known methods, sensitive information can be masked by reducing the resolution of the image. However, this method may affect other image areas that do not need to be masked, for example, the definition of other image areas may be reduced, and the quality of the distributed remote sensing image may be affected. Therefore, in order to improve the quality of the distributed remote sensing image, a method for shielding the sensitive information of the remote sensing image is urgently needed.
Disclosure of Invention
The invention provides a remote sensing image sensitive information automatic shielding method and device based on deep learning, and aims to improve the quality of a released remote sensing image. The specific technical scheme is as follows.
In a first aspect, an embodiment of the present invention provides an automatic shielding method for sensitive information of remote sensing images based on deep learning, where the method includes:
acquiring a remote sensing image to be processed;
determining coordinate information of sensitive information contained in the remote sensing image to be processed, and converting pixel values of a region corresponding to the coordinate information in the remote sensing image to be processed into preset pixel values to obtain a candidate remote sensing image;
inputting the candidate remote sensing image into a model obtained by pre-training to obtain a target remote sensing image obtained by pixel filling of a preset pixel value area in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of a preset area of each sample remote sensing image into the preset pixel values according to each sample remote sensing image.
Optionally, the training process of the model includes:
constructing an initial model; the initial model comprises a full convolution fill sub-network and a recognizer;
obtaining sample remote sensing images, and converting pixel values of preset areas of the sample remote sensing images into preset pixel values to obtain candidate sample remote sensing images;
inputting each sample remote sensing image and each corresponding candidate sample remote sensing image into the initial model, filling each candidate sample remote sensing image by the full convolution filling sub-network to obtain a target sample remote sensing image, identifying each sample remote sensing image and each target sample remote sensing image by the identifier, and taking the current initial model as the model when the ratio of the number of the remote sensing images identified as the original image by the identifier to the number of the remote sensing images of the filled images is greater than a preset proportional threshold.
Optionally, the full-convolution padding sub-network includes: a convolutional layer, a pooling layer, a void convolutional layer, and a reverse convolutional layer; the step of obtaining the target sample remote sensing image after filling each candidate sample remote sensing image by the full convolution filling sub-network comprises the following steps:
and after the candidate sample remote sensing images pass through a convolution layer, a pooling layer, a cavity convolution layer and a deconvolution layer, dimension reduction is carried out through a zero-padding deconvolution layer, and the target sample remote sensing image is obtained through up-sampling of the convolution layer.
Optionally, the identifier includes a global identifier and a local identifier; the step of identifying each sample remote sensing image and each target sample remote sensing image by the identifier comprises the following steps:
aiming at each sample remote sensing image, the global recognizer recognizes the sample remote sensing image and determines the sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the sample remote sensing image and determines the remote sensing image as an original image or a filling image; superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the sample remote sensing image;
aiming at each target sample remote sensing image, the global recognizer recognizes the target sample remote sensing image and determines the target sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the target sample remote sensing image and determines the remote sensing image as an original image or a filling image; and superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the remote sensing image of the target sample.
Optionally, the step of determining the coordinate information of the sensitive information contained in the remote sensing image to be processed includes:
acquiring attribute information and coordinate information of each known sensitive information contained in a sensitive information base, and attribute information and coordinate information of each sensitive information contained in the remote sensing image to be processed;
detecting whether candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed or not according to the attribute information and the coordinate information of the known sensitive information and the attribute information and the coordinate information of the sensitive information contained in the remote sensing image to be processed;
if so, determining the coordinate information of the candidate sensitive information as the coordinate information of the sensitive information contained in the remote sensing image to be processed;
if not, inputting the remote sensing image to be processed into a detection network to obtain coordinate information of sensitive information contained in the remote sensing image to be processed; the detection network is obtained by training according to the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information.
Optionally, after the to-be-processed remote sensing image is input into a detection network to obtain coordinate information of sensitive information included in the to-be-processed remote sensing image, the method further includes:
and storing the attribute information and the coordinate information of the sensitive information to the sensitive information library.
Optionally, the preset pixel value is 0 or 255.
In a second aspect, an embodiment of the present invention provides an automatic shielding device for sensitive information of remote sensing images based on deep learning, where the device includes:
the to-be-processed image acquisition module is used for acquiring a to-be-processed remote sensing image;
the coordinate information determining module is used for determining coordinate information of sensitive information contained in the remote sensing image to be processed, and converting pixel values of a region corresponding to the coordinate information in the remote sensing image to be processed into preset pixel values to obtain a candidate remote sensing image;
the sensitive information shielding module is used for inputting the candidate remote sensing image into a model obtained by pre-training to obtain a target remote sensing image after pixel filling is carried out on a preset pixel value area in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of a preset area of each sample remote sensing image into the preset pixel values according to each sample remote sensing image.
Optionally, the apparatus further comprises:
the initial model building module is used for building an initial model; the initial model comprises a full convolution fill sub-network and a recognizer;
the sample image acquisition module is used for acquiring sample remote sensing images, and converting pixel values of preset areas of the sample remote sensing images into the preset pixel values to obtain candidate sample remote sensing images;
the model training module is used for inputting each sample remote sensing image and each corresponding candidate sample remote sensing image into the initial model, the full convolution filling sub-network fills each candidate sample remote sensing image to obtain a target sample remote sensing image, the recognizer recognizes each sample remote sensing image and each target sample remote sensing image, and when the ratio of the number of the remote sensing images recognized as original images by the recognizer to the number of the remote sensing images of the filled images is larger than a preset proportion threshold value, the current initial model is used as the model.
Optionally, the full-convolution padding sub-network includes: a convolutional layer, a pooling layer, a void convolutional layer, and a reverse convolutional layer; the neural network training module is specifically configured to:
and after the candidate sample remote sensing images pass through a convolution layer, a pooling layer, a cavity convolution layer and a deconvolution layer, dimension reduction is carried out through a zero-padding deconvolution layer, and the target sample remote sensing image is obtained through up-sampling of the convolution layer.
Optionally, the identifier includes a global identifier and a local identifier; the neural network training module is specifically configured to:
aiming at each sample remote sensing image, the global recognizer recognizes the sample remote sensing image and determines the sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the sample remote sensing image and determines the remote sensing image as an original image or a filling image; superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the sample remote sensing image;
aiming at each target sample remote sensing image, the global recognizer recognizes the target sample remote sensing image and determines the target sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the target sample remote sensing image and determines the remote sensing image as an original image or a filling image; and superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the remote sensing image of the target sample.
Optionally, the coordinate information determining module includes:
the information acquisition submodule is used for acquiring attribute information and coordinate information of each known sensitive information contained in a sensitive information base and attribute information and coordinate information of each sensitive information contained in the remote sensing image to be processed;
the sensitive information matching sub-module is used for detecting whether candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed or not according to the attribute information and the coordinate information of the known sensitive information and the attribute information and the coordinate information of the sensitive information contained in the remote sensing image to be processed;
the coordinate information determining submodule is used for determining the coordinate information of the candidate sensitive information as the coordinate information of the sensitive information contained in the remote sensing image to be processed when the sensitive information matching submodule detects that the candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed;
the sensitive information detection submodule is used for inputting the remote sensing image to be processed into a detection network when the sensitive information matching submodule does not detect that candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed, so as to obtain coordinate information of the sensitive information contained in the remote sensing image to be processed; the detection network is obtained by training according to the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information.
Optionally, the apparatus further comprises:
and the information storage module is used for storing the attribute information and the coordinate information of the sensitive information to the sensitive information base.
Optionally, the preset pixel value is 0 or 255.
As can be seen from the above, the method and the device for automatically shielding sensitive information of a remote sensing image based on deep learning according to the embodiments of the present invention can remove the sensitive information after determining the coordinate information of the sensitive information in the remote sensing image to be processed, and fill the remote sensing image after removing the sensitive information based on a pre-trained model, that is, only fill a proper picture in the area where the sensitive information is located in the remote sensing image without performing any processing on other areas, thereby not only ensuring that the sensitive information is not disclosed, but also not affecting the image quality of the released remote sensing image. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
The innovation points of the embodiment of the invention comprise:
1. after determining the coordinate information of the sensitive information in the remote sensing image to be processed, the sensitive information is scratched out, and the remote sensing image with the sensitive information scratched out is filled based on a model obtained by pre-training, namely, only a proper picture is filled in the area where the sensitive information is located in the remote sensing image, and no processing is performed on other areas, so that the sensitive information is not disclosed, and the image quality of the issued remote sensing image is not influenced.
2. The candidate sample remote sensing image is an image obtained by cutting out a certain region in the sample remote sensing image, the correlation between the certain region in the remote sensing image and the surrounding region image can be obtained through the sample remote sensing image and a model obtained by training the candidate sample remote sensing image, therefore, when the sensitive information of the remote sensing image is shielded, the model can fill the sensitive information region according to the image characteristics of the surrounding region of the sensitive information region after the candidate remote sensing image obtained by cutting out the sensitive information is input into the model, the complete remote sensing image is obtained, and the image quality of the released remote sensing image is ensured.
3. The attribute information and the coordinate information of the sensitive information identified by the detection network are stored in the sensitive information base, so that the coordinate information of the identified sensitive information can be directly determined according to the sensitive information base in the subsequent sensitive information shielding process, and the sensitive information shielding efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are merely exemplary of some embodiments of the invention. For a person skilled in the art, without inventive effort, further figures can be obtained from these figures.
FIG. 1 is a flowchart of a method for automatically shielding sensitive information of remote sensing images based on deep learning according to an embodiment of the present invention;
FIG. 2 is another flowchart of a method for automatically shielding sensitive information of remote sensing images based on deep learning according to an embodiment of the present invention;
FIG. 3 is another flowchart of a method for automatically shielding sensitive information of remote sensing images based on deep learning according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a model training process according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an automatic shielding device for sensitive information of remote sensing images based on deep learning according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely a few embodiments of the invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a remote sensing image sensitive information automatic shielding method and device based on deep learning, which can improve the quality of a released remote sensing image. The following provides a detailed description of embodiments of the invention.
Fig. 1 is a schematic flow chart of a method for automatically shielding sensitive information of remote sensing images based on deep learning according to an embodiment of the present invention. The method is applied to the electronic equipment. The method specifically comprises the following steps.
S110: and acquiring a remote sensing image to be processed.
The remote sensing image to be processed can be a remote sensing image acquired by a remote sensing satellite. After the remote sensing satellite acquires the remote sensing image, the remote sensing image can be sent to the electronic equipment, and the electronic equipment can take the received remote sensing image as the remote sensing image to be processed.
S120: and determining coordinate information of sensitive information contained in the remote sensing image to be processed, and converting pixel values of a region corresponding to the coordinate information in the remote sensing image to be processed into preset pixel values to obtain a candidate remote sensing image.
The sensitive information is the sensitive information which does not meet the public requirement, in the embodiment of the invention, the sensitive information can also be called as a target, and the remote sensing image can be released only after the sensitive information is shielded. The coordinate information of the sensitive information may be longitude and latitude coordinates thereof.
It can be understood that the remote sensing images shot by the remote sensing satellite are usually marked with coordinate information of each sensitive information. In addition, in some cases, it is known which sensitive information cannot be disclosed in the remote sensing image, and in some cases, the sensitive information that cannot be disclosed in the remote sensing image is unknown.
In the embodiment of the invention, in order to accurately determine the sensitive information in the remote sensing image to be processed, a sensitive information base can be constructed in advance, and the sensitive information base can comprise attribute information and coordinate information of known sensitive information. For example, the sensitive information base may include all known attribute information such as names, colors, purposes and the like which cannot disclose the sensitive information, and longitude and latitude coordinates of each sensitive information.
In one implementation, the process of determining the coordinate information of the sensitive information contained in the remote sensing image to be processed may include the following steps.
S210: and acquiring attribute information and coordinate information of each known sensitive information contained in the sensitive information base and attribute information and coordinate information of each sensitive information contained in the remote sensing image to be processed.
Specifically, the attribute information and the coordinate information of each known sensitive information may be obtained from the corresponding storage location according to the storage location of the sensitive information base. The attribute information of each sensitive information in the remote sensing image to be processed can be obtained by carrying out sensitive information identification on the remote sensing image to be processed; or, the attribute information of each sensitive information can be searched according to the existing data and the coordinate information of each sensitive information in the remote sensing image to be processed.
S220: detecting whether candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed or not according to the attribute information and the coordinate information of the known sensitive information and the attribute information and the coordinate information of the sensitive information contained in the remote sensing image to be processed; if yes, step S230 is performed, and if no, step S240 is performed.
The candidate sensitive information matched with the known sensitive information is the sensitive information which is the same as the attribute information and the coordinate information of the known sensitive information.
S230: and determining the coordinate information of the candidate sensitive information as the coordinate information of the sensitive information contained in the remote sensing image to be processed.
When the candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed, the candidate sensitive information in the remote sensing image to be processed is shown to be the known sensitive information. That is to say, it is known whether the sensitive information contained in the remote sensing image to be processed can be disclosed, and all the sensitive information in the remote sensing image to be processed can be determined only by the sensitive information base.
In this case, the coordinate information of the candidate sensitive information may be directly determined as the coordinate information of the sensitive information included in the remote sensing image to be processed. That is to say, the candidate sensitive information is sensitive information that does not meet public requirements in the remote sensing image to be processed, and the coordinate information of the candidate sensitive information is coordinate information of known sensitive information that is matched with the candidate sensitive information.
S240: inputting the remote sensing image to be processed into a detection network to obtain coordinate information of sensitive information contained in the remote sensing image to be processed; the detection network is obtained by training according to the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information.
When the candidate sensitive information matched with the known sensitive information does not exist in the remote sensing image to be processed, the fact that whether the sensitive information contained in the remote sensing image to be processed can be disclosed or not is unknown is shown, and the sensitive information in the remote sensing image to be processed cannot be determined only through the sensitive information base. In this case, the sensitive information in the remote sensing image to be processed can be detected, and the coordinate information of the sensitive information included in the remote sensing image can be determined.
In the embodiment of the invention, the detection network capable of carrying out two-classification on the sensitive information in the remote sensing image can be obtained by pre-training. That is, the network can identify the sensitive information in the input remote sensing image, and determine that each piece of sensitive information is the sensitive information which does not meet the public standard or the sensitive information which can be disclosed.
Specifically, the detection network may be obtained in advance by training based on the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information. For example, a resnet50 network structure scene classification method can be used for performing classification training to obtain a detection network. The basis for completing the detection network training is that the identification result of the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information is the same as the result of whether the marked remote sensing images contain the sensitive information or not.
After the remote sensing image to be processed is input into the trained detection network, the detection network can perform sliding prediction on the remote sensing image to be processed in a blocking sliding window mode, and the position corresponding to the sensitive information is detected. Specifically, the detection network can perform convolution pooling on the input image, ensure transmission of effective information in the image by connecting shallow and deep network features, and finally output a binary classification result by 50 layers of transmission training.
In one implementation, after identifying the coordinate information of the sensitive information in the remote sensing image to be processed, the attribute information and the coordinate information of the sensitive information can be stored in a sensitive information base. Therefore, in the subsequent sensitive information shielding process, the coordinate information of the identified sensitive information can be directly determined according to the sensitive information base, and the sensitive information shielding efficiency is improved.
After the coordinate information of the sensitive information is determined, the pixel value of the area corresponding to the coordinate information in the remote sensing image to be processed can be converted into a preset pixel value, and a candidate remote sensing image is obtained. The preset pixel value may be 0 or 255. That is, the sensitive information can be scratched out, and the area where the sensitive information is located can be changed into white or black.
S130: inputting the candidate remote sensing image into a model obtained by pre-training to obtain a target remote sensing image obtained by pixel filling of a preset pixel value area in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of preset areas of the sample remote sensing images into preset pixel values according to the sample remote sensing images.
In the embodiment of the invention, the model can be obtained by training according to the remote sensing images of the samples and the candidate remote sensing images obtained after the pixel values of the preset area of the remote sensing images of the samples are converted into the preset pixel values. The candidate sample remote sensing image is an image obtained by cutting out a certain region in the sample remote sensing image, and the correlation between the certain region in the remote sensing image and the surrounding region image can be obtained through the sample remote sensing image and a model obtained by training the candidate sample remote sensing image.
When the sensitive information of the remote sensing image is shielded, after the candidate remote sensing image with the sensitive information removed is input into the model, the model can fill the sensitive information region according to the image characteristics of the surrounding region of the sensitive information region to obtain a target remote sensing image, namely a complete remote sensing image, after pixel filling is carried out on the preset pixel value region in the candidate remote sensing image, so that the image quality of the released remote sensing image is ensured.
As can be seen from the above, the method for automatically shielding sensitive information of remote sensing images based on deep learning provided by the embodiment of the present invention can remove the sensitive information after determining the coordinate information of the sensitive information in the remote sensing image to be processed, and fill the remote sensing image after removing the sensitive information based on the model obtained by pre-training, that is, only fill a proper picture in the area where the sensitive information is located in the remote sensing image, and do not perform any processing on other areas, thereby not only ensuring that the sensitive information is not disclosed, but also not affecting the image quality of the released remote sensing image.
As an implementation manner of the embodiment of the present invention, as shown in fig. 3, the training process of the model may include:
s310: constructing an initial model; the initial model includes a full convolution fill sub-network and a recognizer.
S320: and obtaining sample remote sensing images, and converting pixel values of preset areas of the sample remote sensing images into preset pixel values to obtain candidate sample remote sensing images.
Each sample remote sensing image may be an image acquired by a remote sensing satellite. The sample remote sensing images may include sensitive information that does not meet public requirements, and may also include sensitive information that can be public, which is not limited in the embodiments of the present invention.
The pixel value of the preset area of each sample remote sensing image is converted into the preset pixel value, namely the content of the preset area of each sample remote sensing image is removed, and an image with the preset area as a single pixel value is obtained and can be called as a candidate sample remote sensing image.
S330: inputting each sample remote sensing image and each corresponding candidate sample remote sensing image into an initial model, filling each candidate sample remote sensing image by a full-convolution filling sub-network to obtain a target sample remote sensing image, identifying each sample remote sensing image and each target sample remote sensing image by an identifier, and taking the current initial model as the model when the ratio of the number of the remote sensing images identified as the original image by the identifier to the number of the remote sensing images of the filled images is greater than a preset proportional threshold.
Specifically, the process of filling each candidate sample remote sensing image by the full convolution filling sub-network may be that after each candidate sample remote sensing image passes through the convolution layer, the pooling layer, the void convolution layer and the deconvolution layer, dimension reduction is performed by the zero-padding deconvolution layer, and the target sample remote sensing image is obtained by upsampling the convolution layer.
The process of identifying the sample remote sensing image and the target sample remote sensing image by the identifier can be that for each sample remote sensing image, the global identifier identifies the sample remote sensing image and determines the sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of a preset area in the sample remote sensing image, namely recognizes the area with the content removed, and determines the area as an original image or a filling image; and superposing the output results of the global recognizer and the local recognizer to obtain the recognition result of the sample remote sensing image.
Aiming at each target sample remote sensing image, the global recognizer recognizes the target sample remote sensing image and determines the target sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of a preset area in the remote sensing image of the target sample, namely recognizes the area with the content removed, and determines the area as an original image or a filling image; and superposing the output results of the global recognizer and the local recognizer to obtain the recognition result of the remote sensing image of the target sample.
In a specific implementation, as shown in fig. 4, a schematic diagram of a model training process in an embodiment of the present invention is shown.
After the sample remote sensing image 410 is obtained, the sample remote sensing image may be copied, and a 0 value is randomly filled in the copied image, so as to obtain a candidate sample remote sensing image 420. Pairs of images can then be placed into the constructed initial model for training.
Aiming at the characteristics of large map range, unfixed ground objects and the like of the high-definition remote sensing image, the sample remote sensing image and the candidate sample remote sensing image can be read into the network for training in a mode of overlapping area and unfixed window size. Wherein, the network structure adopts convolution kernels with different scales.
The model in the embodiment of the invention comprises two parts, wherein the first part is a full convolution filling sub-network 430, the input candidate sample remote sensing image is subjected to convolution, pooling and cavity convolution, dimension reduction is performed by zero padding deconvolution, and finally, a new image is formed by convolution and up-sampling, and the new image can be called as a target sample remote sensing image.
The second part is the recognizer 440, which includes a global recognizer and a local recognizer, the global recognizer inputs the size of the whole image, the local recognizer inputs the region of 0 complement, and after multi-layer convolution pooling, the outputs of the two are finally concatenated, the output result is predicted, and whether the image is an actual image (true) or a filled image (false) is determined.
The candidate sample remote sensing image is an image obtained by cutting out a certain region in the sample remote sensing image, the correlation between the certain region in the remote sensing image and the surrounding region image can be obtained through the sample remote sensing image and a model obtained by training the candidate sample remote sensing image, therefore, when the sensitive information of the remote sensing image is shielded, the model can fill the sensitive information region according to the image characteristics of the surrounding region of the sensitive information region after the candidate remote sensing image obtained by cutting out the sensitive information is input into the model, the complete remote sensing image is obtained, and the image quality of the released remote sensing image is ensured.
As shown in fig. 5, an automatic shielding device for sensitive information of remote sensing images based on deep learning according to an embodiment of the present invention may include:
a to-be-processed image obtaining module 510, configured to obtain a to-be-processed remote sensing image;
a coordinate information determining module 520, configured to determine coordinate information of sensitive information included in the remote sensing image to be processed, and convert a pixel value of a region corresponding to the coordinate information in the remote sensing image to be processed into a preset pixel value to obtain a candidate remote sensing image;
the sensitive information shielding module 530 is configured to input the candidate remote sensing image into a model obtained through pre-training, so as to obtain a target remote sensing image obtained by pixel filling of a preset pixel value region in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of a preset area of each sample remote sensing image into the preset pixel values according to each sample remote sensing image.
Optionally, the apparatus further comprises:
the initial network building module is used for building an initial model; the initial model comprises a full convolution fill sub-network and a recognizer;
the sample image acquisition module is used for acquiring sample remote sensing images, and converting pixel values of preset areas of the sample remote sensing images into the preset pixel values to obtain candidate sample remote sensing images;
the model training module is used for inputting each sample remote sensing image and each corresponding candidate sample remote sensing image into the initial model, the full convolution filling sub-network fills each candidate sample remote sensing image to obtain a target sample remote sensing image, the recognizer recognizes each sample remote sensing image and each target sample remote sensing image, and when the ratio of the number of the remote sensing images recognized as original images by the recognizer to the number of the remote sensing images of the filled images is larger than a preset proportion threshold value, the current initial model is used as the model.
Optionally, the full-convolution padding sub-network includes: a convolutional layer, a pooling layer, a void convolutional layer, and a reverse convolutional layer; the neural network training module is specifically configured to:
and after the candidate sample remote sensing images pass through a convolution layer, a pooling layer, a cavity convolution layer and a deconvolution layer, dimension reduction is carried out through a zero-padding deconvolution layer, and the target sample remote sensing image is obtained through up-sampling of the convolution layer.
Optionally, the identifier includes a global identifier and a local identifier; the neural network training module is specifically configured to:
aiming at each sample remote sensing image, the global recognizer recognizes the sample remote sensing image and determines the sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the sample remote sensing image and determines the remote sensing image as an original image or a filling image; superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the sample remote sensing image;
aiming at each target sample remote sensing image, the global recognizer recognizes the target sample remote sensing image and determines the target sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the target sample remote sensing image and determines the remote sensing image as an original image or a filling image; and superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the remote sensing image of the target sample.
Optionally, the coordinate information determining module 520 includes:
the information acquisition submodule is used for acquiring attribute information and coordinate information of each known sensitive information contained in a sensitive information base and attribute information and coordinate information of each sensitive information contained in the remote sensing image to be processed;
the sensitive information matching sub-module is used for detecting whether candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed or not according to the attribute information and the coordinate information of the known sensitive information and the attribute information and the coordinate information of the sensitive information contained in the remote sensing image to be processed;
the coordinate information determining submodule is used for determining the coordinate information of the candidate sensitive information as the coordinate information of the sensitive information contained in the remote sensing image to be processed when the sensitive information matching submodule detects that the candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed;
the sensitive information detection submodule is used for inputting the remote sensing image to be processed into a detection network when the sensitive information matching submodule does not detect that candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed, so as to obtain coordinate information of the sensitive information contained in the remote sensing image to be processed; the detection network is obtained by training according to the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information.
Optionally, the apparatus further comprises:
and the information storage module is used for storing the attribute information and the coordinate information of the sensitive information to the sensitive information base.
Optionally, the preset pixel value is 0 or 255.
As can be seen from the above, the automatic shielding device for sensitive information of remote sensing images based on deep learning provided in the embodiments of the present invention can remove the sensitive information after determining the coordinate information of the sensitive information in the remote sensing image to be processed, and fill the remote sensing image after removing the sensitive information based on the model obtained by pre-training, that is, only fill a proper picture in the area where the sensitive information is located in the remote sensing image, and do not perform any processing on other areas, thereby not only ensuring that the sensitive information is not disclosed, but also not affecting the image quality of the remote sensing image that is released.
The above device embodiment corresponds to the method embodiment, and has the same technical effect as the method embodiment, and for the specific description, refer to the method embodiment. The device embodiment is obtained based on the method embodiment, and for specific description, reference may be made to the method embodiment section, which is not described herein again.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A remote sensing image sensitive information automatic shielding method based on deep learning is characterized by comprising the following steps:
acquiring a remote sensing image to be processed;
determining coordinate information of sensitive information contained in the remote sensing image to be processed, and converting pixel values of a region corresponding to the coordinate information in the remote sensing image to be processed into preset pixel values to obtain a candidate remote sensing image;
inputting the candidate remote sensing image into a model obtained by pre-training to obtain a target remote sensing image obtained by pixel filling of a preset pixel value area in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of a preset area of each sample remote sensing image into the preset pixel values according to each sample remote sensing image.
2. The method of claim 1, wherein the training process of the model comprises:
constructing an initial model; the initial model comprises a full convolution fill sub-network and a recognizer;
obtaining sample remote sensing images, and converting pixel values of preset areas of the sample remote sensing images into preset pixel values to obtain candidate sample remote sensing images;
inputting each sample remote sensing image and each corresponding candidate sample remote sensing image into the initial model, filling each candidate sample remote sensing image by the full convolution filling sub-network to obtain a target sample remote sensing image, identifying each sample remote sensing image and each target sample remote sensing image by the identifier, and taking the current initial model as the model when the ratio of the number of the remote sensing images identified as the original image by the identifier to the number of the remote sensing images of the filled images is greater than a preset proportional threshold.
3. The method of claim 2, wherein the fully convolved fill subnetwork comprises: a convolutional layer, a pooling layer, a void convolutional layer, and a reverse convolutional layer; the step of obtaining the target sample remote sensing image after filling each candidate sample remote sensing image by the full convolution filling sub-network comprises the following steps:
and after the candidate sample remote sensing images pass through a convolution layer, a pooling layer, a cavity convolution layer and a deconvolution layer, dimension reduction is carried out through a zero-padding deconvolution layer, and the target sample remote sensing image is obtained through up-sampling of the convolution layer.
4. The method of claim 2, wherein the recognizer comprises a global recognizer and a local recognizer; the step of identifying each sample remote sensing image and each target sample remote sensing image by the identifier comprises the following steps:
aiming at each sample remote sensing image, the global recognizer recognizes the sample remote sensing image and determines the sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the sample remote sensing image and determines the remote sensing image as an original image or a filling image; superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the sample remote sensing image;
aiming at each target sample remote sensing image, the global recognizer recognizes the target sample remote sensing image and determines the target sample remote sensing image as an original image or a filling image; the local recognizer recognizes the remote sensing image of the preset area in the target sample remote sensing image and determines the remote sensing image as an original image or a filling image; and superposing the output results of the global recognizer and the local recognizer to be used as the recognition result of the remote sensing image of the target sample.
5. The method of claim 1, wherein the step of determining coordinate information of sensitive information contained in the remote sensing image to be processed comprises:
acquiring attribute information and coordinate information of each known sensitive information contained in a sensitive information base, and attribute information and coordinate information of each sensitive information contained in the remote sensing image to be processed;
detecting whether candidate sensitive information matched with the known sensitive information exists in the remote sensing image to be processed or not according to the attribute information and the coordinate information of the known sensitive information and the attribute information and the coordinate information of the sensitive information contained in the remote sensing image to be processed;
if so, determining the coordinate information of the candidate sensitive information as the coordinate information of the sensitive information contained in the remote sensing image to be processed;
if not, inputting the remote sensing image to be processed into a detection network to obtain coordinate information of sensitive information contained in the remote sensing image to be processed; the detection network is obtained by training according to the remote sensing image containing the sensitive information and the remote sensing image not containing the sensitive information.
6. The method according to claim 5, wherein after inputting the remote sensing image to be processed into a detection network and obtaining coordinate information of sensitive information contained in the remote sensing image to be processed, the method further comprises:
and storing the attribute information and the coordinate information of the sensitive information to the sensitive information library.
7. The method according to any one of claims 1-6, wherein the predetermined pixel value is 0 or 255.
8. The device for automatically shielding the sensitive information of the remote sensing image based on the deep learning is characterized by comprising the following components:
the to-be-processed image acquisition module is used for acquiring a to-be-processed remote sensing image;
the coordinate information determining module is used for determining coordinate information of sensitive information contained in the remote sensing image to be processed, and converting pixel values of a region corresponding to the coordinate information in the remote sensing image to be processed into preset pixel values to obtain a candidate remote sensing image;
the sensitive information shielding module is used for inputting the candidate remote sensing image into a model obtained by pre-training to obtain a target remote sensing image after pixel filling is carried out on a preset pixel value area in the candidate remote sensing image; the model is obtained by training candidate sample remote sensing images obtained by converting pixel values of a preset area of each sample remote sensing image into the preset pixel values according to each sample remote sensing image.
9. The apparatus of claim 8, further comprising:
the initial model building module is used for building an initial model; the initial model comprises a full convolution fill sub-network and a recognizer;
the sample image acquisition module is used for acquiring sample remote sensing images, and converting pixel values of preset areas of the sample remote sensing images into the preset pixel values to obtain candidate sample remote sensing images;
the model training module is used for inputting each sample remote sensing image and each corresponding candidate sample remote sensing image into the initial model, the full convolution filling sub-network fills each candidate sample remote sensing image to obtain a target sample remote sensing image, the recognizer recognizes each sample remote sensing image and each target sample remote sensing image, and when the ratio of the number of the remote sensing images recognized as original images by the recognizer to the number of the remote sensing images of the filled images is larger than a preset proportion threshold value, the current initial model is used as the model.
10. The apparatus of claim 9, wherein the full convolution fill sub-network comprises: a convolutional layer, a pooling layer, a void convolutional layer, and a reverse convolutional layer; the neural network training module is specifically configured to:
and after the candidate sample remote sensing images pass through a convolution layer, a pooling layer, a cavity convolution layer and a deconvolution layer, dimension reduction is carried out through a zero-padding deconvolution layer, and the target sample remote sensing image is obtained through up-sampling of the convolution layer.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911329520.6A CN110992257B (en) | 2019-12-20 | 2019-12-20 | Remote sensing image sensitive information automatic shielding method and device based on deep learning |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911329520.6A CN110992257B (en) | 2019-12-20 | 2019-12-20 | Remote sensing image sensitive information automatic shielding method and device based on deep learning |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110992257A true CN110992257A (en) | 2020-04-10 |
CN110992257B CN110992257B (en) | 2024-06-14 |
Family
ID=70073787
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911329520.6A Active CN110992257B (en) | 2019-12-20 | 2019-12-20 | Remote sensing image sensitive information automatic shielding method and device based on deep learning |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110992257B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112732912A (en) * | 2020-12-30 | 2021-04-30 | 平安科技(深圳)有限公司 | Sensitive tendency expression detection method, device, equipment and storage medium |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170076438A1 (en) * | 2015-08-31 | 2017-03-16 | Cape Analytics, Inc. | Systems and methods for analyzing remote sensing imagery |
WO2017071160A1 (en) * | 2015-10-28 | 2017-05-04 | 深圳大学 | Sea-land segmentation method and system for large-size remote-sensing image |
CN108109126A (en) * | 2018-01-12 | 2018-06-01 | 适普远景遥感信息技术(北京)有限公司 | A kind of target area filling and method for amalgamation processing based on satellite remote-sensing image |
WO2018214195A1 (en) * | 2017-05-25 | 2018-11-29 | 中国矿业大学 | Remote sensing imaging bridge detection method based on convolutional neural network |
CN109255334A (en) * | 2018-09-27 | 2019-01-22 | 中国电子科技集团公司第五十四研究所 | Remote sensing image terrain classification method based on deep learning semantic segmentation network |
CN109271972A (en) * | 2018-11-05 | 2019-01-25 | 常熟理工学院 | Intelligent image identifying system and method based on natural language understanding and image graphics |
CN109446992A (en) * | 2018-10-30 | 2019-03-08 | 苏州中科天启遥感科技有限公司 | Remote sensing image building extracting method and system, storage medium, electronic equipment based on deep learning |
CN109934200A (en) * | 2019-03-22 | 2019-06-25 | 南京信息工程大学 | A kind of RGB color remote sensing images cloud detection method of optic and system based on improvement M-Net |
CN109961068A (en) * | 2017-12-26 | 2019-07-02 | 阿里巴巴集团控股有限公司 | Image recognition, training, searching method and device and equipment, medium |
CN110175538A (en) * | 2019-05-10 | 2019-08-27 | 国网福建省电力有限公司龙岩供电公司 | A kind of substation's Bird's Nest recognition methods and system based on machine learning |
CN110287869A (en) * | 2019-06-25 | 2019-09-27 | 吉林大学 | High-resolution remote sensing image Crop classification method based on deep learning |
CN110399819A (en) * | 2019-07-15 | 2019-11-01 | 北京洛斯达数字遥感技术有限公司 | A kind of remote sensing image residential block extraction method based on deep learning |
-
2019
- 2019-12-20 CN CN201911329520.6A patent/CN110992257B/en active Active
Patent Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170076438A1 (en) * | 2015-08-31 | 2017-03-16 | Cape Analytics, Inc. | Systems and methods for analyzing remote sensing imagery |
WO2017071160A1 (en) * | 2015-10-28 | 2017-05-04 | 深圳大学 | Sea-land segmentation method and system for large-size remote-sensing image |
WO2018214195A1 (en) * | 2017-05-25 | 2018-11-29 | 中国矿业大学 | Remote sensing imaging bridge detection method based on convolutional neural network |
CN109961068A (en) * | 2017-12-26 | 2019-07-02 | 阿里巴巴集团控股有限公司 | Image recognition, training, searching method and device and equipment, medium |
CN108109126A (en) * | 2018-01-12 | 2018-06-01 | 适普远景遥感信息技术(北京)有限公司 | A kind of target area filling and method for amalgamation processing based on satellite remote-sensing image |
CN109255334A (en) * | 2018-09-27 | 2019-01-22 | 中国电子科技集团公司第五十四研究所 | Remote sensing image terrain classification method based on deep learning semantic segmentation network |
CN109446992A (en) * | 2018-10-30 | 2019-03-08 | 苏州中科天启遥感科技有限公司 | Remote sensing image building extracting method and system, storage medium, electronic equipment based on deep learning |
CN109271972A (en) * | 2018-11-05 | 2019-01-25 | 常熟理工学院 | Intelligent image identifying system and method based on natural language understanding and image graphics |
CN109934200A (en) * | 2019-03-22 | 2019-06-25 | 南京信息工程大学 | A kind of RGB color remote sensing images cloud detection method of optic and system based on improvement M-Net |
CN110175538A (en) * | 2019-05-10 | 2019-08-27 | 国网福建省电力有限公司龙岩供电公司 | A kind of substation's Bird's Nest recognition methods and system based on machine learning |
CN110287869A (en) * | 2019-06-25 | 2019-09-27 | 吉林大学 | High-resolution remote sensing image Crop classification method based on deep learning |
CN110399819A (en) * | 2019-07-15 | 2019-11-01 | 北京洛斯达数字遥感技术有限公司 | A kind of remote sensing image residential block extraction method based on deep learning |
Non-Patent Citations (1)
Title |
---|
刘文涛;李世华;覃驭楚;: "基于全卷积神经网络的建筑物屋顶自动提取", no. 11 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112732912A (en) * | 2020-12-30 | 2021-04-30 | 平安科技(深圳)有限公司 | Sensitive tendency expression detection method, device, equipment and storage medium |
CN112732912B (en) * | 2020-12-30 | 2024-04-09 | 平安科技(深圳)有限公司 | Sensitive trend expression detection method, device, equipment and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN110992257B (en) | 2024-06-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111738110A (en) | Remote sensing image vehicle target detection method based on multi-scale attention mechanism | |
CN109086668B (en) | Unmanned aerial vehicle remote sensing image road information extraction method based on multi-scale generation countermeasure network | |
CN111222395A (en) | Target detection method and device and electronic equipment | |
CN111914843B (en) | Character detection method, system, equipment and storage medium | |
CN111310770B (en) | Target detection method and device | |
US10373316B2 (en) | Images background subtraction for dynamic lighting scenarios | |
CN112487848A (en) | Character recognition method and terminal equipment | |
CN110020658B (en) | Salient object detection method based on multitask deep learning | |
CN114926766A (en) | Identification method and device, equipment and computer readable storage medium | |
CN111523558A (en) | Ship shielding detection method and device based on electronic purse net and electronic equipment | |
CN110992257B (en) | Remote sensing image sensitive information automatic shielding method and device based on deep learning | |
CN117765485A (en) | Vehicle type recognition method, device and equipment based on improved depth residual error network | |
CN111179245B (en) | Image quality detection method, device, electronic equipment and storage medium | |
CN111339808A (en) | Vehicle collision probability prediction method and device, electronic equipment and storage medium | |
CN112001258B (en) | Method, device, equipment and storage medium for identifying on-time arrival of logistics truck | |
CN111881996A (en) | Object detection method, computer device and storage medium | |
CN116453086A (en) | Method and device for identifying traffic sign and electronic equipment | |
CN112861979B (en) | Trademark identification method, trademark identification device, computing equipment and computer storage medium | |
CN113298102B (en) | Training method and device for target classification model | |
CN115393247A (en) | Image processing method, base station feeder line detection method and related device | |
CN113326891A (en) | Method, system and device for detecting small target object | |
CN108805190B (en) | Image processing method and device | |
CN117934486B (en) | Transformer element detection method, device, electronic equipment and storage medium | |
EP4398205A1 (en) | Image anti-spoofing detection method and apparatus | |
CN116596867B (en) | Target detection method for welding image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |