CN109543606A - A kind of face identification method that attention mechanism is added - Google Patents

A kind of face identification method that attention mechanism is added Download PDF

Info

Publication number
CN109543606A
CN109543606A CN201811396296.8A CN201811396296A CN109543606A CN 109543606 A CN109543606 A CN 109543606A CN 201811396296 A CN201811396296 A CN 201811396296A CN 109543606 A CN109543606 A CN 109543606A
Authority
CN
China
Prior art keywords
face
attention mechanism
network
image
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201811396296.8A
Other languages
Chinese (zh)
Other versions
CN109543606B (en
Inventor
郑伟诗
叶海佳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National Sun Yat Sen University
Original Assignee
National Sun Yat Sen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National Sun Yat Sen University filed Critical National Sun Yat Sen University
Priority to CN201811396296.8A priority Critical patent/CN109543606B/en
Publication of CN109543606A publication Critical patent/CN109543606A/en
Application granted granted Critical
Publication of CN109543606B publication Critical patent/CN109543606B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172Classification, e.g. identification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168Feature extraction; Face representation

Abstract

The invention discloses a kind of face identification methods that attention mechanism is added, Face datection and face registration process first are carried out to data set with cascade neural network, the deep neural network that attention mechanism is added is constructed again, test sample input is finally trained attention mechanism network and carries out recognition of face by training attention mechanism network.The present invention constructs attention mechanism using STN module, and different STN modules is input to after each stage of deep neural network, the output result fusion of the series connection output result and deep neural network of STN module is got up, as output feature.In order to which network can adaptively be learnt to the region of interest characteristic of field with judgement index, the present invention uses the method for carrying out affine transformation operation to input by STN module, strengthen understanding and study of the network to local message, on existing recognition of face network, the accuracy rate for improving recognition of face enhances the robustness of identifying system.

Description

A kind of face identification method that attention mechanism is added
Technical field
The present invention relates to machine deep learning, image procossing identification field more particularly to a kind of addition attention mechanism Face identification method.
Background technique
Recognition of face is one of project most challenging in computer vision field and machine learning field in recent years, by The effective recognition of face of extensive concern success for having arrived researchers has broad application prospects, can be in national defense safety, video The scenes such as monitoring, human-computer interaction and video index are played a great role.
Currently, the feature extraction Web vector graphic Classification Loss (Softmax Loss) for being mostly based on CNN is instructed as network Experienced supervisory signals, these networks are the distance between different classes of in the training process gradually to increase to be classified as learning objective Greatly.Deepface uses sorter network method, while using complicated 3D alignment thereof and a large amount of training data.DeepID is then It is that piecemeal is carried out to face picture first, feature extraction then is carried out to different faces block using multiple sorter networks, is finally made With joint bayesian algorithm these features are merged, due to the technology be to different faces block carry out feature extraction, so Data set increases several times than original image, and the training time greatly increases, and computing resource consumption is big.In addition these face blocks are all tight Lattice fix division mode, and for side face or irregular face picture, then the accuracy rate can have a greatly reduced quality, and algorithm is inadequate Robust.
Summary of the invention
In order to overcome the shortcomings of the prior art, the present invention provides a kind of recognition of face side that attention mechanism is added Method, by paying attention to power module, neural network can learn automatically to the face block feature with identification, rather than fixed partition Face block is more advantageous to promotion classification accuracy with the feature that such method is extracted, and robustness is stronger.Simultaneously because paying attention to Power module is simple for structure, so computing resource consumption is few, network convergence rate is fast.
In order to achieve the above object, the invention adopts the following technical scheme:
The present invention discloses a kind of face identification method that attention mechanism is added, and includes the following steps:
S1: image preprocessing, the facial image being aligned are carried out using cascade convolutional neural networks;
S2: data augmentation is carried out to pretreated image, the data augmentation includes random cropping and random overturning behaviour Make, go out the size area set by step S1 treated image random cropping, image is overturn with the probability of setting, Whitening processing finally is done to image, the image being sized then directly is normalized into for test sample, is then carried out at albefaction Reason, it is described be sized it is identical as being sized for random cropping;
S3: setting attention mechanism module learns to the face block feature with identification automatically for network, utilizes note Power mechanism module anticipate for the image progress convolution operation of input, then carries out connecting to return entirely exporting M angle value, M is nature Number constructs matrix based on M angle value, the local feature of image is extracted by matrix operation;
S4: building attention mechanism network, extracts characteristics of image using deep neural network, and attention mechanism mould is added Block, the attention mechanism network include main road and branch, and the main road is that picture is defeated by obtaining after deep neural network Out, the branch is that different attention mechanism modules is passed through in the output in each stage of deep neural network, then successively carries out The output of main road and branch is finally carried out merging features, obtains final image by the output obtained after elementwise-add Characteristic pattern, for calculating loss function and as the feature of recognition of face;
S5: training attention mechanism network is trained simultaneously attention mechanism network using recognition of face loss function And it saves;
S6: characteristics of image is extracted, test sample is input in trained attention mechanism network, good figure is obtained As feature;
S7: recognition of face classifies the characteristics of image that extraction obtains with softmax homing method, completes test specimens This identification.
As a preferred technical solution, cascade convolutional neural networks described in step S1 use MTCNN, including P-Net, R-Net and O-Net gives any one testing image, zooms to different proportion, constructs image pyramid, then sequentially inputs P-Net, R-Net and O-Net extract face candidate frame, further include that fitting face and non-face classification, frame recurrence and face are special The target training that sign point coordinate returns, specific loss function are as described below:
MTCNN carries out face and non-face classification uses cross entropy as loss function, is denoted as Ldet, calculation formula is as follows:
Wherein, p(i)For the probability of model prediction,For test sample x(i)Label,
MTCNN carries out frame recurrence and uses L2Loss as loss function, is denoted as Lbox, calculation formula is as follows:
Wherein,It is the regressand value of model prediction,It is test sample x(i)True coordinate value, and
MTCNN carries out the recurrence of human face characteristic point coordinate and equally uses L2Loss as loss function, is denoted as Llandmark, calculate Formula is as follows:
Wherein,It is the regressand value of model prediction,It is test sample x(i)The coordinate of real human face characteristic point Value, and
The MTCNN introduces catalogue scalar functions as a preferred technical solution, participates in damage for excluding non-face data The calculating of function is lost, the catalogue scalar functions calculation formula is as follows:
Wherein, N indicates training sample sum, αjIndicate significance level of the corresponding objective function in total objective function, Associated weight for P-Net or R-Net is (αdet=1, αbox=0.5, αlandmark=0.5);For the associated weight of ONet For (αdet=1, αbox=0.5, αlandmark=1).
Attention mechanism module described in step S3 uses STN module, the STN module packet as a preferred technical solution, Localized network module, mesh generator and sampler are included,
The picture of input is carried out convolution operation by the localized network module, is then carried out full connection and is returned out 6 angles Angle value forms the matrix of 2*3,
The mesh generator calculates each position in target figure V by matrix operation and corresponds to coordinate in original image U Position generates Tθ(Gi), specific formula for calculation is as described below:
Wherein,The coordinate of original graph is represented,Represent the coordinate of target figure, AθTo localize net 6 angle values that network module network returns out,
The sampler is sampled in original graph U according to the coordinate information in T (G), and the pixel in U is copied to In target figure V.
As a preferred technical solution, in step S4, the basic network of the deep neural network uses resnet50, Resnet50 includes 5 stage, described in detail below:
Stage0: including convolutional layer and pond layer, the convolution kernel size of the convolutional layer is 7x7, and output channel number is 64, Step-length is 2, and the pond layer uses the pond mode of maxpooling, window size 3x3, step-length 2;
Stage1: it is made of the block that 3 output channel numbers are 256;
Stage2: it is made of the block that 4 output channel numbers are 512;
Stage3: it is made of the block that 5 output channel numbers are 1024;
Stage4: it is made of the block that 6 output channel numbers are 2048;
The characteristics of image figure difference that the branch network obtains the stage0,1,2,3,4 of basic network resnet50 is defeated Enter into each STN module, obtain feature L0, L1, L2, L3, L4, the L1-L4 does a convolution operation, convolution kernel size For 1x1, step-length 1, output channel number is the port number of a upper feature, with the mode of elementwise-add these spies Sign is successively added, specific calculation are as follows:
L0+f(L1)+f(L2)+f(L3)+f(L4)
Wherein "+" it is that elsemenwise-add is operated, f () is convolution operation.
Described piece of structure forming step is described in detail below as a preferred technical solution:
Dimensionality reduction is carried out using 1x1 convolution, then carries out 3x3 convolution operation, then rise dimension with 1x1 convolution, export with it is defeated It is entering to carry out to obtain after elementwise-add operation as a result,
The full articulamentum for being eventually adding one 128 dimension carries out dimensionality reduction.
Recognition of face loss function described in step S5 uses Softmax function as a preferred technical solution, is based on The road K of the disaggregated model of Softmax function exports are as follows:
WhereinbkFor Softmax layers of two parameters, K group weight and biasing are indicated.
Use unactivated full articulamentum for described Softmax layers as a preferred technical solution,.
As a preferred technical solution, after the Softmax layers of output transform K class posterior probability are as follows:
For the maximum probability of each test sample generic, Softmax Loss is defined are as follows:
Wherein θ indicates model parameter, x(i)Indicate test sample y(i)Generic.
The disaggregated model based on Softmax function further includes optimizer as a preferred technical solution, and optimizer is adopted Use Adam.
Compared with the prior art, the invention has the following advantages and beneficial effects:
It (1) is starting point the present invention is based on the face local feature for more having identification is extracted, in the frame of base neural network Attention mechanism module is devised under frame, and is combined with unique connection type and deep neural network, is formd unique The face identification method of attention mechanism is added, the face characteristic of abundant classification relevant information can be extracted.
(2) present invention carries out data augmentation, including random cropping and random turning operation to pretreated image, is used for Increase the sample data of training, the robustness of Strengthens network is capable of in the data amplification of training set.
(3) attention mechanism module of the invention uses STN module, and STN module includes localized network module, and grid is raw It grows up to be a useful person and sampler, the STN modular structure is succinct, and computing resource consumption is few, and network convergence rate is fast.
Detailed description of the invention
Fig. 1 is the structural schematic diagram that face of the present invention is aligned network;
Fig. 2 is the structural schematic diagram of STN module of the present invention;
Fig. 3 is the structural schematic diagram of depth of foundation convolutional neural networks of the present invention;
Fig. 4 is the block structure schematic diagram in depth of foundation convolutional neural networks of the present invention;
Fig. 5 is the structural schematic diagram of attention mechanism network of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and It is not used in the restriction present invention.
The present embodiment discloses a kind of face recognition algorithms based on addition attention mechanism, and the algorithm includes following step It is rapid:
Step 1: the data prediction of Face datection face alignment, the cascade of use are carried out using cascade neural network Convolutional neural networks be MTCNN, MTCNN cascade structure is mainly made of 3 convolutional neural networks, respectively P-Net, R- Net and O-Net.A picture to be detected is given, picture can be first scaled to different ratios, empty with the scale for constructing picture Between, three networks are sequentially input, then to extract face candidate frame.As shown in Figure 1, there are three stage compositions for the algorithm: first Stage, the CNN of shallow-layer quickly generate candidate forms;Second stage refines candidate forms by more complicated CNN, abandons a large amount of Overlapping forms;Phase III, the more powerful CNN of use realize candidate forms going or staying, while showing five facial key points Positioning.When carrying out model training, in order to merge Face datection and face alignment task, MTCNN is fitted 3 mesh simultaneously Mark: face/non-face classification, frame return and human face characteristic point coordinate returns.Three loss functions are respectively:
(1) face/non-face classification
Face/non-face is that two classification problems so MTCNN uses cross entropy as loss function are denoted as Ldet。 For each test sample x(i),
Wherein, p(i)For the probability of model prediction,For test sample x(i)Label,
(2) frame returns: the purpose that frame returns is to estimate and neighbouring real human face area each face candidate frame The offset in domain, including the left side, top, width and height.So frame recurrence be a regression problem, using above-mentioned 4 numerical value as Regressive object is denoted as L so MTCNN uses L2Loss as loss functionbox.For each test sample x(i),
It wherein, is the regressand value of model prediction,It is test sample x(i)True coordinate value, because of target to be returned There are 4 values, so
(3) human face characteristic point coordinate returns
The recurrence of human face characteristic point coordinate is equally a regression problem, since MTCNN only detects 5 human face characteristic points, and Each characteristic point includes x, y-coordinate, so one shares 10 regressive objects.Equally use L2Loss as loss function here, It is denoted as Llandmark.For each test sample x(i):
Wherein,It is the regressand value of model prediction,It is test sample x(i)The coordinate of real human face characteristic point Value, because target to be returned has 10 values,
(4) catalogue scalar functions
It allows and model while being fitted different targets, need using different types of training data, such as non-face picture, portion Divide face picture, band characteristic point mark human face data etc., but not all data are all significant to all objective functions, such as Non-face data are to LlandmarkIt is not significant.It is not that every kind of sample requires to participate in being damaged thus when training The calculating of function is lost, in order to carry out distinguishing to different samples, MTCNN introduces sample type labelIndicate sample x(i)Whether type j is belonged to, and then general objective function representation is
Wherein, N indicates training sample sum, αjIndicate significance level of the corresponding objective function in total objective function, For P-Net and R-Net, associated weight is (αdet=1, αbox=0.5, αlandmark=0.5);And for ONet, in order to guarantee The accuracy of human face characteristic point improves the weight of characteristic point coordinate regressive object function, becomes (αdet=1, αbox=0.5, αlandmark=1)
Step 2: data augmentation
Data augmentation uses random cropping and random turning operation, the former will by step 1 treated picture with Machine cuts out the region 160x160, and the latter overturns picture with 0.5 probability.Albefaction finally is carried out to picture.Test sample It is then directly normalized into the picture of 160x160 size, then equally carries out albefaction.
Step 3: design attention mechanism module
Attention mechanism module is using STN module: as shown in Fig. 2, STN module is by localized network module (Localisation Network), mesh generator (Grid generator), 3 parts of sampler (Sampler) form.
Localisation Network: the network is exactly a simple Recurrent networks.The picture of input is carried out several A convolution operation, then full connection returns out 6 angle values (assuming that being affine transformation), the matrix of 2*3.
Grid generator: mesh generator is responsible for the coordinate position in V by matrix operation, calculating target Each position in figure V corresponds to the coordinate position in original image U, i.e. generation Tθ(Gi)。
Here Grid sampling process is exactly simple square for two dimensional affine transformation (rotation translates, scaling) Battle array operation:
In above formula,The coordinate of original graph is represented,Represent the coordinate of target figure.AθFor 6 angle values that Localisation Network net regression goes out.
Sampler: sampler is according to Tθ(Gi) in coordinate information, sampled in original graph U, the pixel in U answered It makes in target figure V.
Step 3: attention mechanism network is built
The method that feature extraction uses deep neural network, the basic network of use is resnet50, then this base again Attention mechanism module is added on plinth.And attention mechanism module is using STN module: input feature vector is carried out several convolution Operation, then full connection returns out 6 angle values (assuming that being affine transformation), the matrix of 2*3.Then it inputs multiplied by this matrix It can obtain the significant feature in part.
Network is divided into main road and branch, and main road is the output that picture is obtained by resnet50, and branch is by different The output that elementwise-add is obtained successively is carried out after STN module again.
Main road: resnet50 is made of 5 stages, wherein each stage includes the operation of several convolution sum pondizations.
As shown in figure 3, resnet50 first is divided by output characteristic pattern size, 5 stage, each stage can be divided into The characteristic pattern size of output is all different.
Stage0 has a convolutional layer and pond layer, and convolution kernel size is 7x7, and output channel number is 64, step-length 2.Pond Change using maxpooling, window size 3x3, step-length 2.
Stage1 is made of the block (block) that 3 output channel numbers are 256.
Stage2 is made of the block (block) that 4 output channel numbers are 512.
Stage3 is made of the block (block) that 5 output channel numbers are 1024.
Stage4 is made of the block (block) that 6 output channel numbers are 2048.
As shown in figure 4, wherein the structure of each block is first to carry out dimensionality reduction with a 1x1 convolution, then carry out 3x3 convolution, finally rises dimension with 1x1 convolution again, and output does elementwise-add operation with input, obtains result.
The full articulamentum for being most followed by one 128 dimension carries out information integration.
Branch: the characteristic pattern that stage0,1,2,3,4 is obtained respectively is input in each STN module and obtains respective spy Sign:
Output of the stage0 after STN is L0;
Output of the Stage1 after STN is L1;
Output of the Stage2 after STN is L2;
Output of the Stage3 after STN is L3;
Output of the Stage4 after STN is L4;
As shown in figure 5, remaining feature all does a convolution operation in addition to first feature, convolution kernel size is 1x1, Step-length is 1, output channel number be a upper feature port number, with the mode of elementwise-add these features successively Fusion is got up, so the meaning for doing convolution operation is exactly for changing characteristic dimension, so as to feature phase add operation.Specific addition side Method is as follows:
L0+f(L1)+f(L2)+f(L3)+f(L4)
Wherein "+" it is that elsemenwise-add is operated, f () is convolution operation.
Main road output and branch output can be thus obtained, the output of two-way is finally carried out merging features, is obtained final Feature.This feature, which will be directly used in, calculates loss function and as the feature of recognition of face.
Step 5: training attention mechanism neural network
In the present embodiment, when constructing Softmax disaggregated model, feature is exported and inputs the road K Softmax floor for x by we (being realized using unactivated full articulamentum), to calculate sample about different classes of posterior probabilityWherein K represents classification number Mesh.Softmax layers include two parameters, W and b, and then kth road exportsIt can be expressed as again:
But since the output of full articulamentum is any number, in order to which sample is about different classes of normalization probability, we Need the posterior probability about kth class then obtained to Softmax layers of output transform are as follows:
In the present embodiment, in order to maximize maximum probability of each sample about generic, we can be defined Softmax Loss are as follows:
θ indicates model parameter, x(i)Indicate sample y(i)Generic.
In the present embodiment, optimizer uses Adam, and weight decays to 5e-5, and batch size is 128, average pond layer Output is operated using dropout, and keeping probability is 0.8.Learning rate adjustable strategies are as follows: first using 0.1 as learning rate to training set Then 3 wheel of training is reduced to 0.01 training, 2 wheel, be then reduced to 0.001 training, 2 wheel again, totally 7 wheel.Every classification for having instructed a wheel Model can all be verified on LFW, and finally trained disaggregated model is saved.
Step 6: learn the high-level characteristic and abstract characteristics of image
Characteristics of image is extracted, test sample is input in trained attention mechanism network, good image is obtained Feature.
Step 7: recognition of face
The characteristics of image that extraction obtains is classified with softmax homing method, completes the identification of test sample.
The above embodiment is a preferred embodiment of the present invention, but embodiments of the present invention are not by above-described embodiment Limitation, other any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present invention, It should be equivalent substitute mode, be included within the scope of the present invention.

Claims (10)

1. a kind of face identification method that attention mechanism is added, which is characterized in that include the following steps:
S1: image preprocessing, the facial image being aligned are carried out using cascade convolutional neural networks;
S2: data augmentation is carried out to pretreated image, the data augmentation includes random cropping and random turning operation, warp It crosses step S1 treated that image random cropping goes out the size area set, image is overturn with the probability of setting, finally Whitening processing is done to image, the image being sized then directly is normalized into for test sample, then carries out whitening processing, institute State be sized it is identical as being sized for random cropping;
S3: setting attention mechanism module learns to the face block feature with identification automatically for network, utilizes attention The image of input is carried out convolution operation by mechanism module, is then carried out full connection and is returned M angle value of output, M is natural number, base Matrix is constructed in M angle value, the local feature of image is extracted by matrix operation;
S4: building attention mechanism network, extracts characteristics of image using deep neural network, and attention mechanism module is added, The attention mechanism network includes main road and branch, and the main road is that picture passes through the output obtained after deep neural network, The branch is that different attention mechanism modules is passed through in the output in each stage of deep neural network, then successively carries out The output of main road and branch is finally carried out merging features, obtains final image by the output obtained after elementwise-add Characteristic pattern, for calculating loss function and as the feature of recognition of face;
S5: training attention mechanism network is trained and is protected to attention mechanism network using recognition of face loss function It deposits;
S6: extracting characteristics of image, test sample be input in trained attention mechanism network, and it is special to obtain good image Sign;
S7: recognition of face classifies the characteristics of image that extraction obtains with softmax homing method, completes test sample Identification.
2. a kind of face identification method that attention mechanism is added according to claim 1, which is characterized in that in step S1 The cascade convolutional neural networks use MTCNN, including P-Net, R-Net and O-Net, give any one testing image, Different proportion is zoomed to, image pyramid is constructed, then sequentially inputs P-Net, R-Net and O-Net, extracts face candidate frame, Further include the target training for being fitted face and non-face classification, frame recurrence and the recurrence of human face characteristic point coordinate, specifically loses letter Number is as described below:
MTCNN carries out face and non-face classification uses cross entropy as loss function, is denoted as Ldet, calculation formula is as follows:
Wherein, p(i)For the probability of model prediction,For test sample x(i)Label,
MTCNN carries out frame recurrence and uses L2Loss as loss function, is denoted as Lbox, calculation formula is as follows:
Wherein,It is the regressand value of model prediction,It is test sample x(i)True coordinate value, and
MTCNN carries out the recurrence of human face characteristic point coordinate and equally uses L2Loss as loss function, is denoted as Llandmark, calculation formula It is as follows:
Wherein,It is the regressand value of model prediction,It is test sample x(i)The coordinate value of real human face characteristic point, And
3. a kind of face identification method that attention mechanism is added according to claim 2, which is characterized in that described MTCNN introduces catalogue scalar functions, and the calculating of loss function is participated in for excluding non-face data, and the catalogue scalar functions calculate Formula is as follows:
Wherein, N indicates training sample sum, αjSignificance level of the corresponding objective function in total objective function is indicated, for P- The associated weight of Net or R-Net is (αdet=1, αbox=0.5, αlandmark=0.5);Associated weight for ONet is (αdet =1, αbox=0.5, αlandmark=1).
4. a kind of face identification method that attention mechanism is added according to claim 1, which is characterized in that step S3 institute Attention mechanism module is stated using STN module, the STN module includes localized network module, mesh generator and sampler,
The picture of input is carried out convolution operation by the localized network module, is then carried out full connection and is returned out 6 angle values, The matrix of 2*3 is formed,
The mesh generator calculates each position in target figure V by matrix operation and corresponds to coordinate position in original image U, Generate Tθ(Gi), specific formula for calculation is as described below:
Wherein,The coordinate of original graph is represented,Represent the coordinate of target figure, AθFor localized network mould 6 angle values that block net regression goes out,
The sampler is sampled in original graph U according to the coordinate information in T (G), and the pixel in U is copied to target Scheme in V.
5. a kind of face identification method that attention mechanism is added according to claim 1, which is characterized in that step S4 In, it includes 5 stage that the basic network of the deep neural network, which uses resnet50, resnet50, described in detail below:
Stage0: including convolutional layer and pond layer, the convolution kernel size of the convolutional layer is 7x7, and output channel number is 64, step-length It is 2, the pond layer uses the pond mode of maxpooling, window size 3x3, step-length 2;
Stage1: it is made of the block that 3 output channel numbers are 256;
Stage2: it is made of the block that 4 output channel numbers are 512;
Stage3: it is made of the block that 5 output channel numbers are 1024;
Stage4: it is made of the block that 6 output channel numbers are 2048;
The characteristics of image figure that the stage0,1,2,3,4 of basic network resnet50 is obtained is separately input to by the branch network In each STN module, feature L0, L1, L2, L3, L4 are obtained, the L1-L4 does a convolution operation, and convolution kernel size is 1x1, step-length 1, output channel number is the port number of a upper feature, with the mode of elementwise-add these features It is successively added, specific calculation are as follows:
L0+f(L1)+f(L2)+f(L3)+f(L4)
Wherein "+" it is that elsemenwise-add is operated, f () is convolution operation.
6. a kind of face identification method that attention mechanism is added according to claim 5, which is characterized in that described piece Structure forming step is described in detail below:
Dimensionality reduction is carried out using 1x1 convolution, then carries out 3x3 convolution operation, then rise dimension with 1x1 convolution, export and input into Row elementwise-add operation after obtain as a result,
The full articulamentum for being eventually adding one 128 dimension carries out dimensionality reduction.
7. a kind of face identification method that attention mechanism is added according to claim 1, which is characterized in that in step S5 The recognition of face loss function uses Softmax function, and the road K of the disaggregated model based on Softmax function exports are as follows:
WhereinbkFor Softmax layers of two parameters, K group weight and biasing are indicated.
8. a kind of face identification method that attention mechanism is added according to claim 7, which is characterized in that described Softmax layers use unactivated full articulamentum.
9. a kind of face identification method that attention mechanism is added according to claim 8, which is characterized in that described The posterior probability of K class after Softmax layers of output transform are as follows:
For the maximum probability of each test sample generic, Softmax Loss is defined are as follows:
Wherein θ indicates model parameter, x(i)Indicate test sample y(i)Generic.
10. a kind of face identification method that attention mechanism is added according to claim 7, which is characterized in that the base It further include optimizer in the disaggregated model of Softmax function, optimizer uses Adam.
CN201811396296.8A 2018-11-22 2018-11-22 Human face recognition method with attention mechanism Active CN109543606B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811396296.8A CN109543606B (en) 2018-11-22 2018-11-22 Human face recognition method with attention mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811396296.8A CN109543606B (en) 2018-11-22 2018-11-22 Human face recognition method with attention mechanism

Publications (2)

Publication Number Publication Date
CN109543606A true CN109543606A (en) 2019-03-29
CN109543606B CN109543606B (en) 2022-09-27

Family

ID=65849048

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811396296.8A Active CN109543606B (en) 2018-11-22 2018-11-22 Human face recognition method with attention mechanism

Country Status (1)

Country Link
CN (1) CN109543606B (en)

Cited By (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110110642A (en) * 2019-04-29 2019-08-09 华南理工大学 A kind of pedestrian's recognition methods again based on multichannel attention feature
CN110135251A (en) * 2019-04-09 2019-08-16 上海电力学院 A kind of group's image Emotion identification method based on attention mechanism and hybrid network
CN110135243A (en) * 2019-04-02 2019-08-16 上海交通大学 A kind of pedestrian detection method and system based on two-stage attention mechanism
CN110188730A (en) * 2019-06-06 2019-08-30 山东大学 Face datection and alignment schemes based on MTCNN
CN110287846A (en) * 2019-06-19 2019-09-27 南京云智控产业技术研究院有限公司 A kind of face critical point detection method based on attention mechanism
CN110334588A (en) * 2019-05-23 2019-10-15 北京邮电大学 Kinship recognition methods and the device of network are paid attention to based on local feature
CN110378961A (en) * 2019-09-11 2019-10-25 图谱未来(南京)人工智能研究院有限公司 Optimization method, critical point detection method, apparatus and the storage medium of model
CN110458829A (en) * 2019-08-13 2019-11-15 腾讯医疗健康(深圳)有限公司 Image quality control method, device, equipment and storage medium based on artificial intelligence
CN110569905A (en) * 2019-09-10 2019-12-13 江苏鸿信系统集成有限公司 Fine-grained image classification method based on generation of confrontation network and attention network
CN110598022A (en) * 2019-08-05 2019-12-20 华中科技大学 Image retrieval system and method based on robust deep hash network
CN110610129A (en) * 2019-08-05 2019-12-24 华中科技大学 Deep learning face recognition system and method based on self-attention mechanism
CN110633689A (en) * 2019-09-23 2019-12-31 天津天地基业科技有限公司 Face recognition model based on semi-supervised attention network
CN110688938A (en) * 2019-09-25 2020-01-14 江苏省未来网络创新研究院 Pedestrian re-identification method integrated with attention mechanism
CN110781760A (en) * 2019-05-24 2020-02-11 西安电子科技大学 Facial expression recognition method and device based on space attention
CN110796072A (en) * 2019-10-28 2020-02-14 桂林电子科技大学 Target tracking and identity recognition method based on double-task learning
CN110837840A (en) * 2019-11-07 2020-02-25 中国石油大学(华东) Picture feature detection method based on attention mechanism
CN111046781A (en) * 2019-12-09 2020-04-21 华中科技大学 Robust three-dimensional target detection method based on ternary attention mechanism
CN111178183A (en) * 2019-12-16 2020-05-19 深圳市华尊科技股份有限公司 Face detection method and related device
CN111242038A (en) * 2020-01-15 2020-06-05 北京工业大学 Dynamic tongue tremor detection method based on frame prediction network
CN111325161A (en) * 2020-02-25 2020-06-23 四川翼飞视科技有限公司 Method for constructing human face detection neural network based on attention mechanism
CN111339813A (en) * 2019-09-30 2020-06-26 深圳市商汤科技有限公司 Face attribute recognition method and device, electronic equipment and storage medium
CN111563468A (en) * 2020-05-13 2020-08-21 电子科技大学 Driver abnormal behavior detection method based on attention of neural network
CN111582044A (en) * 2020-04-15 2020-08-25 华南理工大学 Face recognition method based on convolutional neural network and attention model
CN111652020A (en) * 2019-04-16 2020-09-11 上海铼锶信息技术有限公司 Method for identifying rotation angle of human face around Z axis
CN111680732A (en) * 2020-05-28 2020-09-18 浙江师范大学 Training method for dish identification based on deep learning attention mechanism
CN111738099A (en) * 2020-05-30 2020-10-02 华南理工大学 Face automatic detection method based on video image scene understanding
CN111783681A (en) * 2020-07-02 2020-10-16 深圳市万睿智能科技有限公司 Large-scale face library recognition method, system, computer equipment and storage medium
CN111860393A (en) * 2020-07-28 2020-10-30 浙江工业大学 Face detection and recognition method on security system
CN111950586A (en) * 2020-07-01 2020-11-17 银江股份有限公司 Target detection method introducing bidirectional attention
CN111967427A (en) * 2020-08-28 2020-11-20 广东工业大学 Fake face video identification method, system and readable storage medium
CN111985323A (en) * 2020-07-14 2020-11-24 珠海市卓轩科技有限公司 Face recognition method and system based on deep convolutional neural network
CN112163462A (en) * 2020-09-08 2021-01-01 北京数美时代科技有限公司 Face-based juvenile recognition method and device and computer equipment
CN112365717A (en) * 2020-10-10 2021-02-12 新疆爱华盈通信息技术有限公司 Vehicle information acquisition method and system
CN112464912A (en) * 2020-12-22 2021-03-09 杭州电子科技大学 Robot-end face detection method based on YOLO-RGGNet
CN112507995A (en) * 2021-02-05 2021-03-16 成都东方天呈智能科技有限公司 Cross-model face feature vector conversion system and method
CN112560756A (en) * 2020-12-24 2021-03-26 北京嘀嘀无限科技发展有限公司 Method, device, electronic equipment and storage medium for recognizing human face
CN112597888A (en) * 2020-12-22 2021-04-02 西北工业大学 On-line education scene student attention recognition method aiming at CPU operation optimization
CN112699847A (en) * 2021-01-15 2021-04-23 苏州大学 Face characteristic point detection method based on deep learning
CN112766422A (en) * 2021-03-15 2021-05-07 山东大学 Privacy protection method based on lightweight face recognition model
CN112766158A (en) * 2021-01-20 2021-05-07 重庆邮电大学 Multi-task cascading type face shielding expression recognition method
WO2021088640A1 (en) * 2019-11-06 2021-05-14 重庆邮电大学 Facial recognition technology based on heuristic gaussian cloud transformation
CN113034457A (en) * 2021-03-18 2021-06-25 广州市索图智能电子有限公司 Face detection device based on FPGA
CN113239866A (en) * 2021-05-31 2021-08-10 西安电子科技大学 Face recognition method and system based on space-time feature fusion and sample attention enhancement
CN113408549A (en) * 2021-07-14 2021-09-17 西安电子科技大学 Few-sample weak and small target detection method based on template matching and attention mechanism
CN113822203A (en) * 2021-09-26 2021-12-21 中国民用航空飞行学院 Face recognition device and method based on reinforcement learning and deep convolutional neural network
CN113971745A (en) * 2021-09-27 2022-01-25 哈尔滨工业大学 Exit-entry verification stamp identification method and device based on deep neural network
CN114943251A (en) * 2022-05-20 2022-08-26 电子科技大学 Unmanned aerial vehicle target identification method based on fusion attention mechanism
CN115993365A (en) * 2023-03-23 2023-04-21 山东省科学院激光研究所 Belt defect detection method and system based on deep learning

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180018539A1 (en) * 2016-07-12 2018-01-18 Beihang University Ranking convolutional neural network constructing method and image processing method and apparatus thereof
CN108009493A (en) * 2017-11-30 2018-05-08 电子科技大学 Face anti-fraud recognition methods based on action enhancing
CN108416314A (en) * 2018-03-16 2018-08-17 中山大学 The important method for detecting human face of picture
CN108537135A (en) * 2018-03-16 2018-09-14 北京市商汤科技开发有限公司 The training method and device of Object identifying and Object identifying network, electronic equipment
CN108564029A (en) * 2018-04-12 2018-09-21 厦门大学 Face character recognition methods based on cascade multi-task learning deep neural network
CN108805089A (en) * 2018-06-14 2018-11-13 南京云思创智信息科技有限公司 Based on multi-modal Emotion identification method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180018539A1 (en) * 2016-07-12 2018-01-18 Beihang University Ranking convolutional neural network constructing method and image processing method and apparatus thereof
CN108009493A (en) * 2017-11-30 2018-05-08 电子科技大学 Face anti-fraud recognition methods based on action enhancing
CN108416314A (en) * 2018-03-16 2018-08-17 中山大学 The important method for detecting human face of picture
CN108537135A (en) * 2018-03-16 2018-09-14 北京市商汤科技开发有限公司 The training method and device of Object identifying and Object identifying network, electronic equipment
CN108564029A (en) * 2018-04-12 2018-09-21 厦门大学 Face character recognition methods based on cascade multi-task learning deep neural network
CN108805089A (en) * 2018-06-14 2018-11-13 南京云思创智信息科技有限公司 Based on multi-modal Emotion identification method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
郑伟诗等: "非对称行人重识别: 跨摄像机持续行人追踪", 《中国科学》 *

Cited By (71)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135243A (en) * 2019-04-02 2019-08-16 上海交通大学 A kind of pedestrian detection method and system based on two-stage attention mechanism
CN110135251A (en) * 2019-04-09 2019-08-16 上海电力学院 A kind of group's image Emotion identification method based on attention mechanism and hybrid network
CN110135251B (en) * 2019-04-09 2023-08-08 上海电力学院 Group image emotion recognition method based on attention mechanism and hybrid network
CN111652020B (en) * 2019-04-16 2023-07-11 上海铼锶信息技术有限公司 Face rotation angle identification method around Z axis
CN111652020A (en) * 2019-04-16 2020-09-11 上海铼锶信息技术有限公司 Method for identifying rotation angle of human face around Z axis
CN110110642A (en) * 2019-04-29 2019-08-09 华南理工大学 A kind of pedestrian's recognition methods again based on multichannel attention feature
CN110334588A (en) * 2019-05-23 2019-10-15 北京邮电大学 Kinship recognition methods and the device of network are paid attention to based on local feature
CN110781760A (en) * 2019-05-24 2020-02-11 西安电子科技大学 Facial expression recognition method and device based on space attention
CN110188730A (en) * 2019-06-06 2019-08-30 山东大学 Face datection and alignment schemes based on MTCNN
CN110188730B (en) * 2019-06-06 2022-12-23 山东大学 MTCNN-based face detection and alignment method
CN110287846A (en) * 2019-06-19 2019-09-27 南京云智控产业技术研究院有限公司 A kind of face critical point detection method based on attention mechanism
CN110287846B (en) * 2019-06-19 2023-08-04 南京云智控产业技术研究院有限公司 Attention mechanism-based face key point detection method
CN110610129A (en) * 2019-08-05 2019-12-24 华中科技大学 Deep learning face recognition system and method based on self-attention mechanism
CN110598022A (en) * 2019-08-05 2019-12-20 华中科技大学 Image retrieval system and method based on robust deep hash network
CN110598022B (en) * 2019-08-05 2021-11-19 华中科技大学 Image retrieval system and method based on robust deep hash network
CN110458829B (en) * 2019-08-13 2024-01-30 腾讯医疗健康(深圳)有限公司 Image quality control method, device, equipment and storage medium based on artificial intelligence
CN110458829A (en) * 2019-08-13 2019-11-15 腾讯医疗健康(深圳)有限公司 Image quality control method, device, equipment and storage medium based on artificial intelligence
CN110569905A (en) * 2019-09-10 2019-12-13 江苏鸿信系统集成有限公司 Fine-grained image classification method based on generation of confrontation network and attention network
CN110569905B (en) * 2019-09-10 2023-04-14 中电鸿信信息科技有限公司 Fine-grained image classification method based on generation of confrontation network and attention network
CN110378961A (en) * 2019-09-11 2019-10-25 图谱未来(南京)人工智能研究院有限公司 Optimization method, critical point detection method, apparatus and the storage medium of model
CN110633689A (en) * 2019-09-23 2019-12-31 天津天地基业科技有限公司 Face recognition model based on semi-supervised attention network
CN110688938A (en) * 2019-09-25 2020-01-14 江苏省未来网络创新研究院 Pedestrian re-identification method integrated with attention mechanism
CN111339813A (en) * 2019-09-30 2020-06-26 深圳市商汤科技有限公司 Face attribute recognition method and device, electronic equipment and storage medium
CN111339813B (en) * 2019-09-30 2022-09-27 深圳市商汤科技有限公司 Face attribute recognition method and device, electronic equipment and storage medium
WO2021063056A1 (en) * 2019-09-30 2021-04-08 深圳市商汤科技有限公司 Facial attribute recognition method and apparatus, and electronic device and storage medium
CN110796072A (en) * 2019-10-28 2020-02-14 桂林电子科技大学 Target tracking and identity recognition method based on double-task learning
CN110796072B (en) * 2019-10-28 2023-04-07 桂林电子科技大学 Target tracking and identity recognition method based on double-task learning
WO2021088640A1 (en) * 2019-11-06 2021-05-14 重庆邮电大学 Facial recognition technology based on heuristic gaussian cloud transformation
CN110837840A (en) * 2019-11-07 2020-02-25 中国石油大学(华东) Picture feature detection method based on attention mechanism
CN111046781B (en) * 2019-12-09 2022-05-27 华中科技大学 Robust three-dimensional target detection method based on ternary attention mechanism
CN111046781A (en) * 2019-12-09 2020-04-21 华中科技大学 Robust three-dimensional target detection method based on ternary attention mechanism
CN111178183A (en) * 2019-12-16 2020-05-19 深圳市华尊科技股份有限公司 Face detection method and related device
CN111242038A (en) * 2020-01-15 2020-06-05 北京工业大学 Dynamic tongue tremor detection method based on frame prediction network
CN111325161A (en) * 2020-02-25 2020-06-23 四川翼飞视科技有限公司 Method for constructing human face detection neural network based on attention mechanism
CN111325161B (en) * 2020-02-25 2023-04-18 四川翼飞视科技有限公司 Method for constructing human face detection neural network based on attention mechanism
CN111582044B (en) * 2020-04-15 2023-06-20 华南理工大学 Face recognition method based on convolutional neural network and attention model
CN111582044A (en) * 2020-04-15 2020-08-25 华南理工大学 Face recognition method based on convolutional neural network and attention model
CN111563468A (en) * 2020-05-13 2020-08-21 电子科技大学 Driver abnormal behavior detection method based on attention of neural network
CN111680732A (en) * 2020-05-28 2020-09-18 浙江师范大学 Training method for dish identification based on deep learning attention mechanism
CN111738099B (en) * 2020-05-30 2023-11-07 华南理工大学 Face automatic detection method based on video image scene understanding
CN111738099A (en) * 2020-05-30 2020-10-02 华南理工大学 Face automatic detection method based on video image scene understanding
CN111950586B (en) * 2020-07-01 2024-01-19 银江技术股份有限公司 Target detection method for introducing bidirectional attention
CN111950586A (en) * 2020-07-01 2020-11-17 银江股份有限公司 Target detection method introducing bidirectional attention
CN111783681A (en) * 2020-07-02 2020-10-16 深圳市万睿智能科技有限公司 Large-scale face library recognition method, system, computer equipment and storage medium
CN111985323A (en) * 2020-07-14 2020-11-24 珠海市卓轩科技有限公司 Face recognition method and system based on deep convolutional neural network
CN111860393A (en) * 2020-07-28 2020-10-30 浙江工业大学 Face detection and recognition method on security system
CN111967427A (en) * 2020-08-28 2020-11-20 广东工业大学 Fake face video identification method, system and readable storage medium
CN112163462A (en) * 2020-09-08 2021-01-01 北京数美时代科技有限公司 Face-based juvenile recognition method and device and computer equipment
CN112365717A (en) * 2020-10-10 2021-02-12 新疆爱华盈通信息技术有限公司 Vehicle information acquisition method and system
CN112464912B (en) * 2020-12-22 2024-02-09 杭州电子科技大学 Robot end face detection method based on YOLO-RGGNet
CN112597888A (en) * 2020-12-22 2021-04-02 西北工业大学 On-line education scene student attention recognition method aiming at CPU operation optimization
CN112597888B (en) * 2020-12-22 2024-03-08 西北工业大学 Online education scene student attention recognition method aiming at CPU operation optimization
CN112464912A (en) * 2020-12-22 2021-03-09 杭州电子科技大学 Robot-end face detection method based on YOLO-RGGNet
CN112560756A (en) * 2020-12-24 2021-03-26 北京嘀嘀无限科技发展有限公司 Method, device, electronic equipment and storage medium for recognizing human face
CN112699847A (en) * 2021-01-15 2021-04-23 苏州大学 Face characteristic point detection method based on deep learning
CN112766158B (en) * 2021-01-20 2022-06-03 重庆邮电大学 Multi-task cascading type face shielding expression recognition method
CN112766158A (en) * 2021-01-20 2021-05-07 重庆邮电大学 Multi-task cascading type face shielding expression recognition method
CN112507995A (en) * 2021-02-05 2021-03-16 成都东方天呈智能科技有限公司 Cross-model face feature vector conversion system and method
CN112507995B (en) * 2021-02-05 2021-06-01 成都东方天呈智能科技有限公司 Cross-model face feature vector conversion system and method
CN112766422A (en) * 2021-03-15 2021-05-07 山东大学 Privacy protection method based on lightweight face recognition model
CN113034457A (en) * 2021-03-18 2021-06-25 广州市索图智能电子有限公司 Face detection device based on FPGA
CN113239866A (en) * 2021-05-31 2021-08-10 西安电子科技大学 Face recognition method and system based on space-time feature fusion and sample attention enhancement
CN113239866B (en) * 2021-05-31 2022-12-13 西安电子科技大学 Face recognition method and system based on space-time feature fusion and sample attention enhancement
CN113408549A (en) * 2021-07-14 2021-09-17 西安电子科技大学 Few-sample weak and small target detection method based on template matching and attention mechanism
CN113408549B (en) * 2021-07-14 2023-01-24 西安电子科技大学 Few-sample weak and small target detection method based on template matching and attention mechanism
CN113822203A (en) * 2021-09-26 2021-12-21 中国民用航空飞行学院 Face recognition device and method based on reinforcement learning and deep convolutional neural network
CN113971745A (en) * 2021-09-27 2022-01-25 哈尔滨工业大学 Exit-entry verification stamp identification method and device based on deep neural network
CN113971745B (en) * 2021-09-27 2024-04-16 哈尔滨工业大学 Method and device for identifying entry-exit check stamp based on deep neural network
CN114943251B (en) * 2022-05-20 2023-05-02 电子科技大学 Unmanned aerial vehicle target recognition method based on fusion attention mechanism
CN114943251A (en) * 2022-05-20 2022-08-26 电子科技大学 Unmanned aerial vehicle target identification method based on fusion attention mechanism
CN115993365A (en) * 2023-03-23 2023-04-21 山东省科学院激光研究所 Belt defect detection method and system based on deep learning

Also Published As

Publication number Publication date
CN109543606B (en) 2022-09-27

Similar Documents

Publication Publication Date Title
CN109543606A (en) A kind of face identification method that attention mechanism is added
CN108537743B (en) Face image enhancement method based on generation countermeasure network
Li et al. Building-a-nets: Robust building extraction from high-resolution remote sensing images with adversarial networks
Tao et al. Smoke detection based on deep convolutional neural networks
CN107038448A (en) Target detection model building method
CN106778604B (en) Pedestrian re-identification method based on matching convolutional neural network
CN105447473B (en) A kind of any attitude facial expression recognizing method based on PCANet-CNN
CN109977918A (en) A kind of target detection and localization optimization method adapted to based on unsupervised domain
CN110503112A (en) A kind of small target deteection of Enhanced feature study and recognition methods
CN108388896A (en) A kind of licence plate recognition method based on dynamic time sequence convolutional neural networks
CN110348376A (en) A kind of pedestrian's real-time detection method neural network based
CN107506722A (en) One kind is based on depth sparse convolution neutral net face emotion identification method
CN109800735A (en) Accurate detection and segmentation method for ship target
CN106096602A (en) A kind of Chinese licence plate recognition method based on convolutional neural networks
CN110458844A (en) A kind of semantic segmentation method of low illumination scene
CN108062551A (en) A kind of figure Feature Extraction System based on adjacency matrix, figure categorizing system and method
CN107392097A (en) A kind of 3 D human body intra-articular irrigation method of monocular color video
Yao et al. Robust CNN-based gait verification and identification using skeleton gait energy image
CN108921879A (en) The motion target tracking method and system of CNN and Kalman filter based on regional choice
CN108280397A (en) Human body image hair detection method based on depth convolutional neural networks
CN107944459A (en) A kind of RGB D object identification methods
CN110246141A (en) It is a kind of based on joint angle point pond vehicles in complex traffic scene under vehicle image partition method
CN109712127A (en) A kind of electric transmission line fault detection method for patrolling video flowing for machine
CN107463881A (en) A kind of character image searching method based on depth enhancing study
Yuan et al. MFFFLD: A multimodal-feature-fusion-based fingerprint liveness detection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant