CN109522807A - Satellite image identifying system, method and electronic equipment based on self-generating feature - Google Patents
Satellite image identifying system, method and electronic equipment based on self-generating feature Download PDFInfo
- Publication number
- CN109522807A CN109522807A CN201811227460.2A CN201811227460A CN109522807A CN 109522807 A CN109522807 A CN 109522807A CN 201811227460 A CN201811227460 A CN 201811227460A CN 109522807 A CN109522807 A CN 109522807A
- Authority
- CN
- China
- Prior art keywords
- layer
- network
- feature
- self
- convolutional layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/469—Contour-based spatial representations, e.g. vector-coding
- G06V10/473—Contour-based spatial representations, e.g. vector-coding using gradient analysis
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Remote Sensing (AREA)
- Astronomy & Astrophysics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
This application involves a kind of satellite image identifying system, method and electronic equipments based on self-generating feature.The system comprises: FCN network struction module: for first of VGG network full articulamentum and second full articulamentum to be become first layer convolutional layer and second layer convolutional layer respectively, construct new FCN network;Self-generating character network constructs module: for being based on GAN network struction self-generating character network;Self-generating feature calculation module: for the self-generating character network to be embedded in the FCN network, the characteristic pattern of input image is generated by the self-generating character network, send the characteristic pattern to the first layer convolutional layer and second layer convolutional layer, the first layer convolutional layer and second layer convolutional layer obtain the recognition result of the input image using the characteristic pattern.The application constantly calculates and updates newest image feature by automatically generating character network, realizes the enhancing of data characteristics and the identification of pixel for pixel.
Description
Technical field
The application belongs to image recognition technology field, in particular to a kind of satellite image based on self-generating feature identifies system
System, method and electronic equipment.
Background technique
With the continuous development of national economy, the utilization of China's land resources is increasingly eager, but accurately to grasp territory money
The land state in source, the algorithm learnt based on traditional feature extracting method and conventional machines cannot be large-scale complex
Land resources image make accurate judgement.
In satellite image identification, the object being accurately partitioned into figure is needed, such as: agricultural land, forest cover, building
Object, river, road, massif etc..Its boundary of exact picture can provide accurate technical parameter for the exploration of next step.It is existing
Technology utilizes traditional Principal Component Analysis, Scale invariant features transform (Scale-invariant feature
Transform, SIFT) method, Haar-like characterization method etc. extract the edge feature of satellite image, to reach recognition effect.
But these features can regard shallow-layer feature as under deep learning, can not capture more effective high-level characteristic, this
Many useful information are just lost, cannot be used in cognitive phase.
At the same time, has 1 [Lin M, Chen Q, Yan S.Network in network [J] .arXiv of document
Preprint arXiv:1312.4400,2013.] it is proved current convolutional neural networks (Convolution Neural
Network, hereinafter referred CNN) last two layers full articulamentum be converted to convolutional layer be not only able to achieve reduce more parameters with
Optimize network, is also able to achieve the recognition effect of pixel for pixel (Pixel to Pixel, hereinafter referred PTP), this is in satellite shadow
As being had a very important significance in accurately identifying.
Another part technology has used CNN network to identify, but CNN network is more suitable for object category
Identification, is not suitable for this intensive identification, that is, the identification of PTP rank, effect can be poor.And operational network occupies GPU
Time is long, and precision is not high.So FCN (Fully Convolutional Network, full convolutional network) network has been used,
But even FCN network can carry out dense-pixel identification, since it was there is no optimization was done to satellite image, not adapt to
In the identification of satellite image.
In addition, a big difficulty that geodata faces is precisely due to satellite image requirement is directed to the identification of pixel, because wanting
Accurately mark off profile, thus picture pretreatment before to carry out artificial element marking, this be it is very time consuming,
Over time, this labeled picture is just very rare, and the picture marked that can really use is very limited.Many works thus
The method that people in the industry utilizes traditional data to expand carries out increase-volume to existing image.But unfortunately, this data extending is brought
Performance boost be very limited, or even under some scenes, the influence of data increase-volume is very little.
Summary of the invention
This application provides a kind of satellite image identifying system, method and electronic equipments based on self-generating feature, it is intended to
One of above-mentioned technical problem in the prior art is solved at least to a certain extent.
To solve the above-mentioned problems, this application provides following technical solutions:
A kind of satellite image identifying system based on self-generating feature, comprising:
FCN network struction module: for becoming first of VGG network full articulamentum and second full articulamentum respectively
First layer convolutional layer and second layer convolutional layer construct new FCN network;
Self-generating character network constructs module: for being based on GAN network struction self-generating character network;
Self-generating feature calculation module: for the self-generating character network to be embedded in the FCN network, by described
Self-generating character network generates the characteristic pattern of input image, sends the characteristic pattern to the first layer convolutional layer and the second layer
Convolutional layer, the first layer convolutional layer and second layer convolutional layer obtain the identification knot of the input image using the characteristic pattern
Fruit.
The technical solution that the embodiment of the present application is taken further includes operator replacement module, and the operator replacement module is used for will
The convolution kernel feature extraction operator of original first convolutional layer replaces with HOG operator in FCN network, passes through the HOG operator
Carry out the Boundary characteristic extraction of the input image.
The technical solution that the embodiment of the present application is taken further include: the mode of the HOG operator extraction boundary characteristic are as follows:
Gradient in different directions is calculated first:
Gx(x, y)=H (x+1, y)-H (x-1, y)
Gy(x, y)=H (x, y+1)-H (x, y-1)
In above-mentioned formula, Gx(x, y) indicates the gradient value of horizontal direction, Gy(x, y) indicates the gradient value of vertical direction, H
(x, y) indicates the gray value that the pixel represents;
Calculate the gradient magnitude of image current pixel point:
Calculate the gradient direction of current pixel point:
The technical solution that the embodiment of the present application is taken further include: the self-generating character network generates satellite image characteristic pattern
Calculation are as follows: after input image, by five convolutional layers original in FCN network carry out image Boundary characteristic extraction,
And boundary characteristic is transferred to self-generating character network, the self-generating character network passes through primary full convolutional layer learning characteristic,
Then one single pass feature of the feature learnt formation is transferred to the second layer convolutional layer;The second layer convolutional layer
Feature and first layer convolutional layer feature common transport to arbiter, minimax process is carried out in arbiter, it is described
Result is fed back to self-generating character network by arbiter again, and the self-generating character network is made to be continuously generated new feature,
After the iterative steps for reaching setting, the newest feature once generated is sent to the second layer convolutional layer, and be transmitted directly to
Full articulamentum, the full articulamentum carry out attribute ballot according to feature, finally obtain the recognition result of input image.
The technical solution that the embodiment of the present application is taken further include: the self-generating character network is embedded into FCN network
Being embedded in point includes:
On the one hand feature after layer 5 convolution sends first layer convolutional layer and second layer convolutional layer to, while by the spy
Sign is transferred to feature generator, this is first insertion point;
The feature that the feature generator is formed is transferred to second layer convolutional layer, this is second insertion point;
In the case where not up to given threshold, the output result of the second layer convolutional layer is only transferred to arbiter, this
Point is embedded in for third.
A kind of another technical solution that the embodiment of the present application is taken are as follows: satellite image identification side based on self-generating feature
Method, comprising the following steps:
Step a: by first of VGG network full articulamentum and second full articulamentum become respectively first layer convolutional layer and
Second layer convolutional layer constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, raw by the self-generating character network
At the characteristic pattern of input image, the characteristic pattern is sent to the first layer convolutional layer and second layer convolutional layer, described first
Layer convolutional layer and second layer convolutional layer using the characteristic pattern obtain the recognition result of the input image.
The technical solution that the embodiment of the present application is taken further include: step a further include: by original first in FCN network
The convolution kernel feature extraction operator of convolutional layer replaces with HOG operator, and the boundary of the input image is carried out by the HOG operator
Feature extraction.
The technical solution that the embodiment of the present application is taken further include: the mode of the HOG operator extraction boundary characteristic are as follows:
Gradient in different directions is calculated first:
Gx(x, y)=H (x+1, y)-H (x-1, y)
Gy(x, y)=H (x, y+1)-H (x, y-1)
In above-mentioned formula, Gx(x, y) indicates the gradient value of horizontal direction, Gy(x, y) indicates the gradient value of vertical direction, H
(x, y) indicates the gray value that the pixel represents;
Calculate the gradient magnitude of image current pixel point:
Calculate the gradient direction of current pixel point:
The technical solution that the embodiment of the present application is taken further include: in the step c, the self-generating character network is generated
The calculation of satellite image characteristic pattern are as follows: after input image, image is carried out by five convolutional layers original in FCN network
Boundary characteristic extraction, and boundary characteristic is transferred to self-generating character network, the self-generating character network is by primary full volume
Then one single pass feature of the feature learnt formation is transferred to the second layer convolutional layer by lamination learning characteristic;Institute
The feature of second layer convolutional layer and the feature common transport of first layer convolutional layer are stated to arbiter, carries out minimum most in arbiter
Result is fed back to self-generating character network by bigization process, the arbiter again, makes the self-generating character network not medium well
The feature of Cheng Xin sends the newest feature once generated to the second layer convolutional layer after reaching the iterative steps of setting,
And it is transmitted directly to full articulamentum, the full articulamentum carries out attribute ballot according to feature, finally obtains the identification of input image
As a result.
The technical solution that the embodiment of the present application is taken further include: in the step c, the self-generating character network insertion
Include: to the insertion point in FCN network
On the one hand feature after layer 5 convolution sends first layer convolutional layer and second layer convolutional layer to, while by the spy
Sign is transferred to feature generator, this is first insertion point;
The feature that the feature generator is formed is transferred to second layer convolutional layer, this is second insertion point;
In the case where not up to given threshold, the output result of the second layer convolutional layer is only transferred to arbiter, this
Point is embedded in for third.
The another technical solution that the embodiment of the present application is taken are as follows: a kind of electronic equipment, comprising:
At least one processor;And
The memory being connect at least one described processor communication;Wherein,
The memory is stored with the instruction that can be executed by one processor, and described instruction is by described at least one
It manages device to execute, so that at least one described processor is able to carry out the above-mentioned satellite image recognition methods based on self-generating feature
Following operation:
Step a: by first of VGG network full articulamentum and second full articulamentum become respectively first layer convolutional layer and
Second layer convolutional layer constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, raw by the self-generating character network
At the characteristic pattern of input image, the characteristic pattern is sent to the first layer convolutional layer and second layer convolutional layer, described first
Layer convolutional layer and second layer convolutional layer using the characteristic pattern obtain the recognition result of the input image.
Compared with the existing technology, the embodiment of the present application generate beneficial effect be: the embodiment of the present application based on spontaneous
At satellite image identifying system, method and the electronic equipment of feature by replacing with the basic convolution nuclear operator in FCN network
Multichannel HOG operator, can four general orientation i.e. up and down diagonal line amount to generated on eight directions it is more sensitive
Detection effect.The full volume specially designed and first, second layer of VGG network full articulamentum is revised as satellite image detection
Product network, while a self-generating character network is added, character network will be automatically generated and be embedded into new full convolutional network,
To constantly calculate and update newest image feature, tie down full convolution net without spending many useless redundant datas of generation
Network.It can not only realize the enhancing of data characteristics (this is also the content of the data of covert expansion), moreover it is possible to realize pixel to picture
The identification of element, to reach the requirement accurately identified.The application avoids expending a large amount of computer resource to realize data extending, energy
Enough requirements for solving data redundancy and exact boundary very well, at the same also particular for appearance less in image before or from
The satellite mapping not occurred optimizes, so that recognizer has more robustness.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of the satellite image identifying system based on self-generating feature of the embodiment of the present application;
Fig. 2 is the FCN network frame figure of the embodiment of the present application;
Fig. 3 is the network frame figure of the satellite image identifying system based on self-generating feature of the embodiment of the present application;
Fig. 4 is the flow chart of the satellite image recognition methods based on self-generating feature of the embodiment of the present application;
Fig. 5 is the hardware device structure of the satellite image recognition methods provided by the embodiments of the present application based on self-generating feature
Schematic diagram.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood
The application is further elaborated.It should be appreciated that specific embodiment described herein is only to explain the application, not
For limiting the application.
Aiming at the problems existing in the prior art, the application generates network (Generative by confrontation
Adversarial Nets, hereinafter referred GAN) [Goodfellow I, Pouget-Abadie J, Mirza M, et
al.Generative adversarial nets[C]//Advances in neural information processing
Systems.2014:2672-2680.] thought constructs a kind of satellite image identifying system based on self-generating feature, automatic raw
The neural network that neotectonics is then utilized at the feature in satellite image, the neural network for automatically generating feature is embedded into new
In full convolutional network, so that it is used to detect the classification of various ground objects in satellite image, can not only realize data characteristics
Enhance (this is also the content of the data of covert expansion), moreover it is possible to realize the identification of pixel for pixel, be accurately identified with reaching
It is required that.
Specifically, referring to Fig. 1, being the knot of the satellite image identifying system based on self-generating feature of the embodiment of the present application
Structure schematic diagram.The satellite image identifying system based on self-generating feature of the embodiment of the present application includes operator replacement module, FCN net
Network constructs module, self-generating character network building module and self-generating feature calculation module.
Operator replacement module: for by the SOBEL operator (Sobel Operator) of the first layer convolutional layer in original FCN network
HOG is replaced with SIFT (Scale Invariant Feature Transform, scale invariant feature conversion) operator
(Histogram of Oriented Gradient, histograms of oriented gradients) operator, remainder layer feature extraction point are constant;Its
In, SOBEL operator can only calculate horizontal and vertical both direction, perform poor when in face of some non-horizontal and vertical direction;
SIFT operator is primarily used to processing Scale invariant, and does not involve too many dimensional variation in satellite image identification,
Only need to consider the detection of image boundary.Horizontal, vertical, four sides in 45 ° of 45 ° of north by east and north by west used herein
To, because of reason symmetrical up and down, i.e. totally eight directions.Since HOG operator extends to multiple directions, this Shen
The convolution kernel feature extraction operator of first layer convolutional layer is please replaced with into HOG operator, it is made to be more in line with the edge of satellite image
Feature detection, can more comprehensively complete the Boundary characteristic extraction of satellite image multiple directions.
In the embodiment of the present application, the direction of HOG operator is determined by the matrix of a 3*3 size, wherein HOG operator calculates
The mode of feature are as follows:
1, gradient in different directions is calculated first:
Gx(x, y)=H (x+1, y)-H (x-1, y) (1)
Gy(x, y)=H (x, y+1)-H (x, y-1) (2)
In formula (1), (2), Gx(x, y) indicates the gradient value of horizontal direction, Gy(x, y) indicates the gradient value of vertical direction,
H (x, y) indicates the gray value that the pixel represents, in image processing, intensity value ranges all in (0,255), wherein 0 represent it is black
Color, 255 indicate white.
(2) gradient magnitude of image current pixel point is calculated:
(3) gradient direction of current pixel point is calculated:
FCN network struction module: for using VGG16 network as bottom frame, by VGG network under the bottom frame
First full articulamentum and second full articulamentum all become convolutional layer (increase newly two convolutional layers), construct new intensive
The FCN network of pixel identification;Wherein, the advantage for increasing two convolutional layers newly is: if only being carried out using a convolutional layer
It samples, then the result up-sampled is likely to be insufficient, is difficult to accomplish precisely to identify in the case where satellite image complexity,
Therefore added one layer of connection, can more preferable characteristic feature, and then accurately indicate the attribute of last pixel.The application is real
The FCN network structure for applying example is described as follows (CONV# represents the convolutional layer number of plies in table 1) shown in table 1:
Port number is the number for representing convolution kernel in table, this is that the important parameter of network is compared with existing VGG network, this
Original two layers full articulamentum is all changed to convolutional layer by application, executes image up-sampling behaviour by two newly-increased convolutional layers
Make, gradual perfection image.What last SOFT-MAX was represented is a kind of classification method common in image.Specifically also referring to
Fig. 2, is the FCN network frame figure of the embodiment of the present application, and the dotted box portion in figure indicates that the application increases structure newly.
Self-generating character network constructs module: for being based on GAN network struction self-generating character network.Wherein, GAN network
It is divided into two parts of generator and arbiter, the basic principle of GAN network are as follows: the effect of generator is from the feature extracted
Study, and then many new images similar with existing image are manufactured, then send the new image of manufacture to arbiter, arbiter
Judge whether image is the picture manufactured, and under the continuous game of two sides, result is fed back to generation by arbiter
Device, generator constantly adjust the study of oneself further according to result, and then produce image more true to nature until arbiter of out-tricking.
The application is based on GAN network, but different is not go to manufacture false picture, but construct self-generating character network by GAN,
Feature more true to nature is generated, the characteristic pattern of more robust is trained, sends these characteristic patterns to up-sampling layer, up-sampling layer will
More accurate satellite image can be identified using these characteristic patterns.
Self-generating character network building mode specifically includes:
In existing x, generator learns a kind of data distribution Pg, simultaneously as there are noise, definition in data distribution
One noise profile function: PZ(Z) guarantee function robust.In addition original parameter θ in networkg, so defined G (z, θg)
For a mapping of legacy data.Arbiter D (x) is used to indicate probability of the data from x, and training D (x) can be maximum
Ability, that is, maximum probability identifies that data come from self training data set or G (x).Simultaneously but also log represented by G
(1-D (G (z))) is minimum, this formula innermost layer nesting is generator, minimum to make the formula, then D (the G of internal layer
(z)) must be maximum, it is such to be meant that arbiter maximization probability accurately identifies the content from generator.By above two
A content combines, and obtains:
Data distribution is become into feature distribution, defining original feature is Of, it is N that generator, which generates feature,f, by formula (5)
Change are as follows:
In formula (6), feature Of、NfIt is a single channel i.e. layer network feature.
Self-generating feature calculation module: for self-generating character network to be embedded into the FCN network of dense-pixel identification,
Generate the satellite image identifying system based on self-generating feature.
It is the satellite image identifying system based on self-generating feature of the embodiment of the present application specifically also referring to Fig. 3
Network frame figure.The satellite image identification method of satellite image identifying system based on self-generating feature specifically: input satellite
After image, firstly, (CONV1 to CONV5) carries out satellite image multiple directions by 5 convolutional layers original in FCN network
Boundary characteristic extraction is then communicated to self-generating character network, and the characteristic pattern of satellite image is generated by self-generating character network,
Send characteristic pattern to up-sampling layer (i.e. newly-increased NEWCONV1 and NEWCONV2), up-sampling layer is obtained using these characteristic patterns
Satellite image recognition result;Wherein, self-generating character network generates the calculation of satellite image characteristic pattern are as follows: original satellite shadow
The feature of picture (CONV5) after layer 5 convolution is transmitted to self-generating character network, and self-generating character network is by primary full convolution
LayerThen the feature learnt is formed a single pass feature (FEATURE by learning characteristic
MAP) it is transferred to newly-increased second layer convolutional layer (NEWCONV2).The feature and first layer of second layer convolutional layer (NEWCONV2) are rolled up
Then the feature common transport of lamination (NEWCONV1) carries out minimax process, arbiter to arbiter in arbiter
Result is fed back to self-generating character network again, it is allowed to be continuously generated the feature that can more mix the spurious with the genuine.Reaching changing for setting
It rides instead of walk after number (i.e. loss function is less than given threshold), newest primary feature is sent to newly-increased second layer convolutional layer
(NEWCONV2), it is then transmitted directly to full articulamentum, full articulamentum carries out attribute ballot according to feature, then obtains satellite shadow
As recognition result.
In the embodiment of the present application, self-generating character network is embedded into the insertion point packet in the FCN network of dense-pixel identification
It includes:
(1), on the one hand the feature after layer 5 convolution sends the newly-increased first layer convolutional layer on original path to
(NEWCONV1) and second layer convolutional layer (NEWCONV2), while this feature is transferred to feature generator, this is embedding for first
Access point.
(2) feature that feature generator is formed is transferred to newly-increased second layer convolutional layer (NEWCONV2), this is second
Insertion point.
(3) it by the output result of second layer convolutional layer (NEWCONV2) in the case where not up to given threshold, only transmits
To arbiter, this is third insertion point.
These three insertion points have taken into account the difference of every layer of feature, have also taken into account the work under arbiter and the mutual game of generator
Use mechanism.
Referring to Fig. 4, being the flow chart of the satellite image recognition methods based on self-generating feature of the embodiment of the present application.This
Apply embodiment the satellite image recognition methods based on self-generating feature the following steps are included:
Step 100: the SOBEL operator of first layer convolutional layer (CONV1) original in FCN network and SIFT operator are replaced
For HOG operator, remainder layer feature extraction point is constant;
In step 100, SOBEL operator can only calculate horizontal and vertical both direction, in face of some non-horizontal and vertical
It performs poor when direction;SIFT operator is primarily used to processing Scale invariant, and does not involve in satellite image identification
Too many dimensional variation, it is only necessary to consider the detection of image boundary.Used herein horizontal, vertical, 45 ° of north by east and north
45 ° of four directions to the west, because of reason symmetrical up and down, i.e. totally eight directions.Due to HOG operator extend to it is multiple
Direction, therefore the convolution kernel feature extraction operator of first layer convolutional layer is replaced with HOG operator by the application, is more in line with it and is defended
The edge feature of star image detects, and can more comprehensively complete the Boundary characteristic extraction of satellite image multiple directions.
In the embodiment of the present application, the direction of HOG operator is determined by the matrix of a 3*3 size, wherein HOG operator calculates
The mode of feature are as follows:
1, gradient in different directions is calculated first:
Gx(x, y)=H (x+1, y)-H (x-1, y) (1)
Gy(x, y)=H (x, y+1)-H (x, y-1) (2)
In formula (1), (2), Gx(x, y) indicates the gradient value of horizontal direction, Gy(x, y) indicates the gradient value of vertical direction,
H (x, y) indicates the gray value that the pixel represents, in image processing, intensity value ranges all in (0,255), wherein 0 represent it is black
Color, 255 indicate white.
(2) gradient magnitude of image current pixel point is calculated:
(3) gradient direction of current pixel point is calculated:
Step 200: using VGG16 network as bottom frame, connect first of VGG network entirely under the bottom frame
Connecing layer and second full articulamentum all becomes convolutional layer (increasing two convolutional layers newly), constructs new dense-pixel identification
FCN network;
In step 200, the advantage of newly-increased two convolutional layers is: if only up-sampled using a convolutional layer,
The result so up-sampled is likely to be insufficient, is difficult to accomplish precisely to identify in the case where satellite image complexity, therefore
Added one layer of connection, can more preferable characteristic feature, and then accurately indicate the attribute of last pixel.The embodiment of the present application
FCN network structure be described as follows (CONV# represents the convolutional layer number of plies in table 1) shown in table 1:
Port number is the number for representing convolution kernel in table, this is that the important parameter of network is compared with existing VGG network, this
Original two layers full articulamentum is all changed to convolutional layer by application, executes image up-sampling behaviour by two newly-increased convolutional layers
Make, gradual perfection image.What last SOFT-MAX was represented is a kind of classification method common in image.
Step 300: being based on GAN network struction self-generating character network;
In step 300, GAN network is divided into two parts of generator and arbiter, the basic principle of GAN network are as follows: generates
The effect of device is to learn from the feature extracted, and then manufacture many new images similar with existing image, then will manufacture
New image send arbiter to, arbiter judges whether image is the picture manufactured, in the continuous game of two sides
Under, result is fed back to generator by arbiter, and generator constantly adjusts the study of oneself further according to result, and then produces more
It is image true to nature until arbiter of out-tricking.The application is based on GAN network, but it is different be do not go to manufacture false picture, and
It is to construct self-generating character network by GAN, generates feature more true to nature, train the characteristic pattern of more robust, by these spy
Sign figure sends up-sampling layer to, and up-sampling layer will identify more accurate satellite image using these characteristic patterns.
Self-generating character network building mode specifically includes:
In existing x, generator learns a kind of data distribution Pg, simultaneously as there are noise, definition in data distribution
One noise profile function: PZ(Z) guarantee function robust.In addition original parameter θ in networkg, so defined G (z, θg)
For a mapping of legacy data.Arbiter D (x) is used to indicate probability of the data from x, and training D (x) can be maximum
Ability, that is, maximum probability identifies that data come from self training data set or G (x).Simultaneously but also log represented by G
(1-D (G (z))) is minimum, this formula innermost layer nesting is generator, minimum to make the formula, then D (the G of internal layer
(z)) must be maximum, it is such to be meant that arbiter maximization probability accurately identifies the content from generator.By above two
A content combines, and obtains:
Data distribution is become into feature distribution, defining original feature is Of, it is N that generator, which generates feature,f, by formula (5)
Change are as follows:
In formula (6), feature Of、NfIt is a single channel i.e. layer network feature.
Step 400: self-generating character network being embedded into the FCN network of dense-pixel identification, generate and be based on self-generating
The satellite image identifying system of feature carries out satellite image identification by the satellite image identifying system based on self-generating feature;
In step 400, the satellite image identification method of the satellite image identifying system based on self-generating feature specifically: base
In the satellite image identification method of the satellite image identifying system of self-generating feature specifically: after input satellite image, firstly, logical
Crossing original 5 convolutional layers in FCN network, (CONV1 to CONV5) carries out the Boundary characteristic extraction of satellite image multiple directions, so
After be transferred to self-generating character network, by self-generating character network generate satellite image characteristic pattern, characteristic pattern is sent to
It up-samples layer (i.e. newly-increased NEWCONV1 and NEWCONV2), up-sampling layer obtains satellite image identification knot using these characteristic patterns
Fruit;Wherein, self-generating character network generates the calculation of satellite image characteristic pattern are as follows: the feature of original satellite image is the 5th
(CONV5) is transmitted to self-generating character network after layer convolution, and self-generating character network is by primary full convolutional layerThen the feature learnt is formed a single pass feature (FEATURE by learning characteristic
MAP) it is transferred to newly-increased second layer convolutional layer (NEWCONV2).The feature and first layer of second layer convolutional layer (NEWCONV2) are rolled up
Then the feature common transport of lamination (NEWCONV1) carries out minimax process, arbiter to arbiter in arbiter
Result is fed back to self-generating character network again, it is allowed to be continuously generated the feature that can more mix the spurious with the genuine.Reaching changing for setting
It rides instead of walk after number (i.e. loss function is less than given threshold), newest primary feature is sent to newly-increased second layer convolutional layer
(NEWCONV2), it is then transmitted directly to full articulamentum, full articulamentum carries out attribute ballot according to feature, then obtains satellite shadow
As recognition result.
Fig. 5 is the hardware device structure of the satellite image recognition methods provided by the embodiments of the present application based on self-generating feature
Schematic diagram.As shown in figure 5, the equipment includes one or more processors and memory.It takes a processor as an example, the equipment
It can also include: input system and output system.
Processor, memory, input system and output system can be connected by bus or other modes, in Fig. 5 with
For being connected by bus.
Memory as a kind of non-transient computer readable storage medium, can be used for storing non-transient software program, it is non-temporarily
State computer executable program and module.Processor passes through operation non-transient software program stored in memory, instruction
And module realizes the place of above method embodiment thereby executing the various function application and data processing of electronic equipment
Reason method.
Memory may include storing program area and storage data area, wherein storing program area can storage program area, extremely
Application program required for a few function;It storage data area can storing data etc..In addition, memory may include that high speed is random
Memory is accessed, can also include non-transient memory, a for example, at least disk memory, flush memory device or other are non-
Transient state solid-state memory.In some embodiments, it includes the memory remotely located relative to processor that memory is optional, this
A little remote memories can pass through network connection to processing system.The example of above-mentioned network includes but is not limited to internet, enterprise
Intranet, local area network, mobile radio communication and combinations thereof.
Input system can receive the number or character information of input, and generate signal input.Output system may include showing
Display screen etc. shows equipment.
One or more of module storages in the memory, are executed when by one or more of processors
When, execute the following operation of any of the above-described embodiment of the method:
Step a: by first of VGG network full articulamentum and second full articulamentum become respectively first layer convolutional layer and
Second layer convolutional layer constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, raw by the self-generating character network
At the characteristic pattern of input image, the characteristic pattern is sent to the first layer convolutional layer and second layer convolutional layer, described first
Layer convolutional layer and second layer convolutional layer using the characteristic pattern obtain the recognition result of the input image.
Method provided by the embodiment of the present application can be performed in the said goods, has the corresponding functional module of execution method and has
Beneficial effect.The not technical detail of detailed description in the present embodiment, reference can be made to method provided by the embodiments of the present application.
The embodiment of the present application provides a kind of non-transient (non-volatile) computer storage medium, and the computer storage is situated between
Matter is stored with computer executable instructions, the executable following operation of the computer executable instructions:
Step a: by first of VGG network full articulamentum and second full articulamentum become respectively first layer convolutional layer and
Second layer convolutional layer constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, raw by the self-generating character network
At the characteristic pattern of input image, the characteristic pattern is sent to the first layer convolutional layer and second layer convolutional layer, described first
Layer convolutional layer and second layer convolutional layer using the characteristic pattern obtain the recognition result of the input image.
The embodiment of the present application provides a kind of computer program product, and the computer program product is non-temporary including being stored in
Computer program on state computer readable storage medium, the computer program include program instruction, when described program instructs
When being computer-executed, the computer is made to execute following operation:
Step a: by first of VGG network full articulamentum and second full articulamentum become respectively first layer convolutional layer and
Second layer convolutional layer constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, raw by the self-generating character network
At the characteristic pattern of input image, the characteristic pattern is sent to the first layer convolutional layer and second layer convolutional layer, described first
Layer convolutional layer and second layer convolutional layer using the characteristic pattern obtain the recognition result of the input image.
Satellite image identifying system, method and the electronic equipment based on self-generating feature of the embodiment of the present application pass through by
Basic convolution nuclear operator in FCN network replaces with multichannel HOG operator, can be right up and down in four general orientation
Linea angulata, which amounts to, generates more sensitive detection effect on eight directions.By the way that first, second layer of VGG network full articulamentum is modified
The full convolutional network specially designed for satellite image detection, while a self-generating character network is added, it will automatically generate
Character network is embedded into new full convolutional network, to constantly calculate and update newest image feature, generates without spending
Many useless redundant datas tie down full convolutional network.It can not only realize that (this is also covert expansion for the enhancing of data characteristics
The content for the data filled), moreover it is possible to the identification of pixel for pixel is realized, to reach the requirement accurately identified.The application avoids expending
A large amount of computer resource realizes data extending, can solve the requirement of data redundancy and exact boundary, while also spy very well
Safety pin to appearance less in image before or the never satellite mapping that occurs optimize so that recognizer has more robust
Property.
The foregoing description of the disclosed embodiments makes professional and technical personnel in the field can be realized or use the application.
Various modifications to these embodiments will be readily apparent to those skilled in the art, defined herein
General Principle can realize in other embodiments without departing from the spirit or scope of the application.Therefore, this Shen
These embodiments shown in the application please be not intended to be limited to, and are to fit to special with principle disclosed in the present application and novelty
The consistent widest scope of point.
Claims (11)
1. a kind of satellite image identifying system based on self-generating feature characterized by comprising
FCN network struction module: for first of VGG network full articulamentum and second full articulamentum to be become first respectively
Layer convolutional layer and second layer convolutional layer, construct new FCN network;
Self-generating character network constructs module: for being based on GAN network struction self-generating character network;
Self-generating feature calculation module: for the self-generating character network to be embedded in the FCN network, by described spontaneous
The characteristic pattern that input image is generated at character network, sends the characteristic pattern to the first layer convolutional layer and second layer convolution
Layer, the first layer convolutional layer and second layer convolutional layer obtain the recognition result of the input image using the characteristic pattern.
2. the satellite image identifying system according to claim 1 based on self-generating feature, which is characterized in that further include calculating
Sub- replacement module, the operator replacement module are used for the convolution kernel feature extraction of first convolutional layer original in FCN network
Operator replaces with HOG operator, and the Boundary characteristic extraction of the input image is carried out by the HOG operator.
3. the satellite image identifying system according to claim 2 based on self-generating feature, which is characterized in that the HOG
The mode of operator extraction boundary characteristic are as follows:
Gradient in different directions is calculated first:
Gx(x, y)=H (x+1, y)-H (x-1, y)
Gy(x, y)=H (x, y+1)-H (x, y-1)
In above-mentioned formula, Gx(x, y) indicates the gradient value of horizontal direction, Gy(x, y) indicates the gradient value of vertical direction, H (x, y)
Indicate the gray value that the pixel represents;
Calculate the gradient magnitude of image current pixel point:
Calculate the gradient direction of current pixel point:
4. the satellite image identifying system according to claim 3 based on self-generating feature, which is characterized in that described spontaneous
The calculation of satellite image characteristic pattern is generated at character network are as follows: after input image, pass through five volumes original in FCN network
Lamination carries out the Boundary characteristic extraction of image, and boundary characteristic is transferred to self-generating character network, the self-generating feature net
Then feature learn is formed a single pass feature and is transferred to described the by network by primary full convolutional layer learning characteristic
Two layers of convolutional layer;The feature of the second layer convolutional layer and the feature common transport of first layer convolutional layer are differentiating to arbiter
Minimax process is carried out in device, result is fed back to self-generating character network again, makes the self-generating by the arbiter
Character network is continuously generated new feature, after reaching the iterative steps of setting, sends the newest feature once generated to institute
Second layer convolutional layer is stated, and is transmitted directly to full articulamentum, the full articulamentum carries out attribute ballot according to feature, finally obtains
The recognition result of input image.
5. the satellite image identifying system according to claim 4 based on self-generating feature, which is characterized in that described spontaneous
Being embedded into the insertion point in FCN network at character network includes:
On the one hand feature after layer 5 convolution sends first layer convolutional layer and second layer convolutional layer to, while this feature being passed
It is defeated by feature generator, this is first insertion point;
The feature that the feature generator is formed is transferred to second layer convolutional layer, this is second insertion point;
In the case where not up to given threshold, the output result of the second layer convolutional layer is only transferred to arbiter, this is
Three insertion points.
6. a kind of satellite image recognition methods based on self-generating feature, which comprises the following steps:
Step a: first of VGG network full articulamentum and second full articulamentum are become into first layer convolutional layer and second respectively
Layer convolutional layer, constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, is generated by the self-generating character network defeated
The characteristic pattern for entering image sends the characteristic pattern to the first layer convolutional layer and second layer convolutional layer, the first layer volume
Lamination and second layer convolutional layer obtain the recognition result of the input image using the characteristic pattern.
7. the satellite image recognition methods according to claim 6 based on self-generating feature, which is characterized in that step a is also
It include: that the convolution kernel feature extraction operator of first convolutional layer original in FCN network is replaced with into HOG operator, by described
HOG operator carries out the Boundary characteristic extraction of the input image.
8. the satellite image recognition methods according to claim 7 based on self-generating feature, which is characterized in that the HOG
The mode of operator extraction boundary characteristic are as follows:
Gradient in different directions is calculated first:
Gx(x, y)=H (x+1, y)-H (x-1, y)
Gy(x, y)=H (x, y+1)-H (x, y-1)
In above-mentioned formula, Gx(x, y) indicates the gradient value of horizontal direction, Gy(x, y) indicates the gradient value of vertical direction, H (x, y)
Indicate the gray value that the pixel represents;
Calculate the gradient magnitude of image current pixel point:
Calculate the gradient direction of current pixel point:
9. the satellite image recognition methods according to claim 8 based on self-generating feature, which is characterized in that in the step
In rapid c, the self-generating character network generates the calculation of satellite image characteristic pattern are as follows: after input image, passes through FCN network
In original five convolutional layers carry out the Boundary characteristic extraction of image, and boundary characteristic is transferred to self-generating character network, institute
Self-generating character network is stated by primary full convolutional layer learning characteristic, the feature learnt is then formed into a single pass spy
Sign is transferred to the second layer convolutional layer;The feature of the second layer convolutional layer and the feature common transport of first layer convolutional layer are given
Arbiter carries out minimax process in arbiter, and result is fed back to self-generating character network by the arbiter again,
So that the self-generating character network is continuously generated new feature, after reaching the iterative steps of setting, is once generated newest
Feature sends the second layer convolutional layer to, and is transmitted directly to full articulamentum, and the full articulamentum carries out attribute according to feature
Ballot, finally obtains the recognition result of input image.
10. the satellite image recognition methods according to claim 9 based on self-generating feature, which is characterized in that described
In step c, the self-generating character network is embedded into the insertion point in FCN network and includes:
On the one hand feature after layer 5 convolution sends first layer convolutional layer and second layer convolutional layer to, while this feature being passed
It is defeated by feature generator, this is first insertion point;
The feature that the feature generator is formed is transferred to second layer convolutional layer, this is second insertion point;
In the case where not up to given threshold, the output result of the second layer convolutional layer is only transferred to arbiter, this is
Three insertion points.
11. a kind of electronic equipment, comprising:
At least one processor;And
The memory being connect at least one described processor communication;Wherein,
The memory is stored with the instruction that can be executed by one processor, and described instruction is by least one described processor
It executes, so that at least one described processor is able to carry out above-mentioned 6 to 10 described in any item satellites based on self-generating feature
The following operation of image recognition method:
Step a: first of VGG network full articulamentum and second full articulamentum are become into first layer convolutional layer and second respectively
Layer convolutional layer, constructs new FCN network;
Step b: it is based on GAN network struction self-generating character network;
Step c: the self-generating character network is embedded in the FCN network, is generated by the self-generating character network defeated
The characteristic pattern for entering image sends the characteristic pattern to the first layer convolutional layer and second layer convolutional layer, the first layer volume
Lamination and second layer convolutional layer obtain the recognition result of the input image using the characteristic pattern.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811227460.2A CN109522807B (en) | 2018-10-22 | 2018-10-22 | Satellite image recognition system and method based on self-generated features and electronic equipment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811227460.2A CN109522807B (en) | 2018-10-22 | 2018-10-22 | Satellite image recognition system and method based on self-generated features and electronic equipment |
Publications (2)
Publication Number | Publication Date |
---|---|
CN109522807A true CN109522807A (en) | 2019-03-26 |
CN109522807B CN109522807B (en) | 2020-11-27 |
Family
ID=65772178
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201811227460.2A Active CN109522807B (en) | 2018-10-22 | 2018-10-22 | Satellite image recognition system and method based on self-generated features and electronic equipment |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN109522807B (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110866908A (en) * | 2019-11-12 | 2020-03-06 | 腾讯科技(深圳)有限公司 | Image processing method, image processing apparatus, server, and storage medium |
CN112132155A (en) * | 2019-06-25 | 2020-12-25 | 珠海市一微半导体有限公司 | SIFT algorithm image boundary expansion method and processing method based on hardware circuit |
CN112446266A (en) * | 2019-09-04 | 2021-03-05 | 北京君正集成电路股份有限公司 | Face recognition network structure suitable for front end |
CN114648706A (en) * | 2022-05-23 | 2022-06-21 | 四川嘉普信工程技术咨询有限公司 | Forest tree species identification method, device and equipment based on satellite remote sensing image |
US11379722B2 (en) | 2019-05-23 | 2022-07-05 | Htc Corporation | Method for training generative adversarial network (GAN), method for generating images by using GAN, and computer readable storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105631415A (en) * | 2015-12-25 | 2016-06-01 | 中通服公众信息产业股份有限公司 | Video pedestrian recognition method based on convolution neural network |
CN107016677A (en) * | 2017-03-24 | 2017-08-04 | 北京工业大学 | A kind of cloud atlas dividing method based on FCN and CNN |
CN108009472A (en) * | 2017-10-25 | 2018-05-08 | 五邑大学 | A kind of finger back arthrosis line recognition methods based on convolutional neural networks and Bayes classifier |
CN108537807A (en) * | 2017-03-02 | 2018-09-14 | 中国科学院遥感与数字地球研究所 | One kind enhancing detection method based on the improved edge of No. four satellite images of high score |
-
2018
- 2018-10-22 CN CN201811227460.2A patent/CN109522807B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105631415A (en) * | 2015-12-25 | 2016-06-01 | 中通服公众信息产业股份有限公司 | Video pedestrian recognition method based on convolution neural network |
CN108537807A (en) * | 2017-03-02 | 2018-09-14 | 中国科学院遥感与数字地球研究所 | One kind enhancing detection method based on the improved edge of No. four satellite images of high score |
CN107016677A (en) * | 2017-03-24 | 2017-08-04 | 北京工业大学 | A kind of cloud atlas dividing method based on FCN and CNN |
CN108009472A (en) * | 2017-10-25 | 2018-05-08 | 五邑大学 | A kind of finger back arthrosis line recognition methods based on convolutional neural networks and Bayes classifier |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11379722B2 (en) | 2019-05-23 | 2022-07-05 | Htc Corporation | Method for training generative adversarial network (GAN), method for generating images by using GAN, and computer readable storage medium |
TWI772805B (en) * | 2019-05-23 | 2022-08-01 | 宏達國際電子股份有限公司 | Method for training generative adversarial network, method for generating image, and computer-readable storage medium |
CN112132155A (en) * | 2019-06-25 | 2020-12-25 | 珠海市一微半导体有限公司 | SIFT algorithm image boundary expansion method and processing method based on hardware circuit |
CN112132155B (en) * | 2019-06-25 | 2022-06-14 | 珠海一微半导体股份有限公司 | SIFT algorithm image boundary expansion method and processing method based on hardware circuit |
CN112446266A (en) * | 2019-09-04 | 2021-03-05 | 北京君正集成电路股份有限公司 | Face recognition network structure suitable for front end |
CN112446266B (en) * | 2019-09-04 | 2024-03-29 | 北京君正集成电路股份有限公司 | Face recognition network structure suitable for front end |
CN110866908A (en) * | 2019-11-12 | 2020-03-06 | 腾讯科技(深圳)有限公司 | Image processing method, image processing apparatus, server, and storage medium |
CN114648706A (en) * | 2022-05-23 | 2022-06-21 | 四川嘉普信工程技术咨询有限公司 | Forest tree species identification method, device and equipment based on satellite remote sensing image |
Also Published As
Publication number | Publication date |
---|---|
CN109522807B (en) | 2020-11-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109522807A (en) | Satellite image identifying system, method and electronic equipment based on self-generating feature | |
JP6976270B2 (en) | Remote determination of the amount stored in a container in a geographic area | |
Kong et al. | A federated learning-based license plate recognition scheme for 5G-enabled internet of vehicles | |
US10157479B2 (en) | Synthesizing training data for broad area geospatial object detection | |
US8379994B2 (en) | Digital image analysis utilizing multiple human labels | |
US11507777B2 (en) | Labeling techniques for a modified panoptic labeling neural network | |
CN103177458B (en) | A kind of visible remote sensing image region of interest area detecting method based on frequency-domain analysis | |
CN106709486A (en) | Automatic license plate identification method based on deep convolutional neural network | |
Zhang et al. | Curriculum-style local-to-global adaptation for cross-domain remote sensing image segmentation | |
CN108776777A (en) | The recognition methods of spatial relationship between a kind of remote sensing image object based on Faster RCNN | |
CN108229274A (en) | Multilayer neural network model training, the method and apparatus of roadway characteristic identification | |
Lou et al. | Preliminary investigation on single remote sensing image inpainting through a modified GAN | |
JP2019185787A (en) | Remote determination of containers in geographical region | |
Khryashchev et al. | Deep learning for region detection in high-resolution aerial images | |
Stomberg et al. | Jungle-net: Using explainable machine learning to gain new insights into the appearance of wilderness in satellite imagery | |
Li et al. | An aerial image segmentation approach based on enhanced multi-scale convolutional neural network | |
CN111310690A (en) | Forest fire recognition method and device based on CN and three-channel capsule network | |
CN104915641B (en) | The method that facial image light source orientation is obtained based on Android platform | |
Wu et al. | Real-time vehicle color recognition based on yolo9000 | |
Yu et al. | Automatic segmentation of golden pomfret based on fusion of multi-head self-attention and channel-attention mechanism | |
CN110751163B (en) | Target positioning method and device, computer readable storage medium and electronic equipment | |
Li et al. | An improved PCB defect detector based on feature pyramid networks | |
CN116824485A (en) | Deep learning-based small target detection method for camouflage personnel in open scene | |
CN116543325A (en) | Unmanned aerial vehicle image-based crop artificial intelligent automatic identification method and system | |
Miao et al. | Use of colour transformation and the geodesic method for road centreline extraction from VHR satellite images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |