CN113486688A - Face recognition method and intelligent device - Google Patents
Face recognition method and intelligent device Download PDFInfo
- Publication number
- CN113486688A CN113486688A CN202010460440.0A CN202010460440A CN113486688A CN 113486688 A CN113486688 A CN 113486688A CN 202010460440 A CN202010460440 A CN 202010460440A CN 113486688 A CN113486688 A CN 113486688A
- Authority
- CN
- China
- Prior art keywords
- image
- face
- cartoon
- face feature
- loss value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000000605 extraction Methods 0.000 claims description 62
- 238000012549 training Methods 0.000 claims description 58
- 230000008569 process Effects 0.000 claims description 22
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000009466 transformation Effects 0.000 claims description 12
- 230000001815 facial effect Effects 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 9
- 238000012795 verification Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 6
- 239000013598 vector Substances 0.000 description 6
- 210000000887 face Anatomy 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 230000003042 antagnostic effect Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 210000001508 eye Anatomy 0.000 description 2
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/80—2D [Two Dimensional] animation, e.g. using sprites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a face recognition method and intelligent equipment, wherein a corresponding cartoon type face image is generated by converting a face image obtained by recognition, so that no matter face information is registered or face recognition is carried out, real face information of a user cannot be stored, personal information of the user can be effectively protected, and the fact that some mechanisms capable of breaking face recognition acquire the real face image or the real face characteristic information of the user is avoided.
Description
Technical Field
The invention relates to the technical field of face recognition, in particular to a face recognition method and intelligent equipment.
Background
Face recognition is a biometric technology for identity recognition based on facial feature information of a person. The technology is convenient in identity recognition and is fast in popularization. At present, various smart homes use face recognition technology. The face recognition technology can facilitate our lives, the trouble of password input is avoided, but the face recognition technology has the risk of information leakage, at present, the phenomena of attacking and breaking a face recognition mechanism through an anti-neural network exist, even if face characteristic information is stored in face recognition instead of a face picture, the anti-neural network can restore the face characteristic information into an original picture, the face recognition is one of biological recognition, the face recognition has uniqueness, once the risk of information leakage is large, huge hidden dangers exist, and how to protect the face information of a user under the condition of not influencing the advantages of a face recognition algorithm is a problem which needs to be solved at present.
Disclosure of Invention
The invention provides a face recognition method and intelligent equipment, which are used for solving the problem of how to protect face information of a user under the condition of not influencing the advantages of a face recognition algorithm.
According to a first aspect of embodiments of the present application, there is provided a smart device, including:
a display configured to display a screen;
an image collector configured to: collecting a reference image containing face information of a user;
a processor configured to:
carrying out face recognition on the reference image collected by the image collector to obtain a face image; generating a cartoon type target image based on the face image; extracting the face feature information of the target image, and comparing the face feature information with at least one face feature information stored in advance; and verifying the reference image according to the comparison result.
Optionally, the processor is configured to:
converting the face image into a target image of a cartoon type based on the trained generated confrontation network model;
and extracting the face feature information of the target image based on the trained face feature extraction model.
Optionally, the processor is configured to:
the method comprises the steps of obtaining a first sample set comprising a plurality of original face images and corresponding original cartoon images, training an initially generated confrontation network model for a plurality of times based on the first sample set, and obtaining a trained generated confrontation network model, wherein each training process comprises the following steps:
selecting a first sample from the first sample set;
inputting the first sample into an initially generated confrontation network model, determining a first loss value based on a cartoon image output by the initially generated confrontation network model and a corresponding original cartoon image, and performing parameter adjustment on the initially generated confrontation network model according to the first loss value;
inputting the output cartoon image and the corresponding original face image into an image generation model, determining a second loss value based on the real face image and the original face image output by the image generation model, and performing parameter adjustment on the image generation model according to the second loss value;
and determining a third loss value according to the original face image and the real face image, and performing parameter adjustment on the initially generated confrontation network model according to the third loss value.
Optionally, the processor is configured to:
acquiring a second sample set comprising a plurality of cartoon images, and training the initial human face feature extraction model for a plurality of times based on the second sample set to obtain a trained human face feature extraction model, wherein each training process comprises the following steps:
selecting at least one cartoon image of the same user from the second sample set, and extracting face feature information of the at least one cartoon image of the same user based on an initial face feature extraction model;
selecting at least one cartoon image of another user from the second sample set, and extracting face feature information of the at least one cartoon image of the another user based on an initial face feature extraction model;
determining a fourth loss value based on the face feature information of the at least one cartoon image of the same user extracted by the initial face feature extraction model and the face feature information of the at least one cartoon image of another user;
and adjusting parameters of the initial face feature extraction model according to the fourth loss value.
Optionally, the processor is configured to:
obtaining an affine transformation matrix based on the face key point information of the face image and the face key point information of a preset standard face image;
and correcting the face image based on the affine transformation matrix.
According to a second aspect of the embodiments of the present application, there is provided a face recognition method, including:
acquiring a reference image containing user face information to perform face recognition to obtain a face image;
generating a cartoon type target image based on the face image;
extracting the face feature information of the target image, and comparing the face feature information with at least one face feature information stored in advance;
and verifying the reference image according to the comparison result.
Optionally, generating a cartoon-type target image based on the face image includes:
converting the face image into a target image of a cartoon type based on the trained generated confrontation network model;
extracting the face feature information of the target image, including:
and extracting the face feature information of the target image based on the trained face feature extraction model.
Optionally, the generating an antagonistic network model is generated by the following training mode:
the method comprises the steps of obtaining a first sample set comprising a plurality of original face images and corresponding original cartoon images, training an initially generated confrontation network model for a plurality of times based on the first sample set, and obtaining a trained generated confrontation network model, wherein each training process comprises the following steps:
selecting a first sample from the first sample set;
inputting the first sample into an initially generated confrontation network model, determining a first loss value based on a cartoon image output by the initially generated confrontation network model and a corresponding original cartoon image, and performing parameter adjustment on the initially generated confrontation network model according to the first loss value;
inputting the output cartoon image and the corresponding original face image into an image generation model, determining a second loss value based on the real face image and the original face image output by the image generation model, and performing parameter adjustment on the image generation model according to the second loss value;
and determining a third loss value according to the original face image and the real face image, and performing parameter adjustment on the initially generated confrontation network model according to the third loss value.
Optionally, the face feature extraction model is generated by the following training method:
acquiring a second sample set comprising a plurality of cartoon images, and training the initial human face feature extraction model for a plurality of times based on the second sample set to obtain a trained human face feature extraction model, wherein each training process comprises the following steps:
selecting at least one cartoon image of the same user from the second sample set, and extracting face feature information of the at least one cartoon image of the same user based on an initial face feature extraction model;
selecting at least one cartoon image of another user from the second sample set, and extracting face feature information of the at least one cartoon image of the another user based on an initial face feature extraction model;
determining a fourth loss value based on the face feature information of the at least one cartoon image of the same user extracted by the initial face feature extraction model and the face feature information of the at least one cartoon image of another user;
and adjusting parameters of the initial face feature extraction model according to the fourth loss value.
Optionally, before generating the cartoon-type target image based on the face image, the method includes:
obtaining an affine transformation matrix based on the face key point information of the face image and the face key point information of a preset standard face image;
and correcting the face image based on the affine transformation matrix.
According to a third aspect of the embodiments of the present application, a chip is provided, where the chip is coupled to a storage unit in a user equipment, so that the chip invokes, when running, program instructions stored in the storage unit, so as to implement the above aspects of the embodiments of the present application and any method that may be involved in the aspects.
According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium storing program instructions, which, when executed on a computer, cause the computer to perform the above aspects of the embodiments of the present application and any of the methods that the aspects relate to.
According to a fifth aspect of embodiments of the present application, there is provided a computer program product, which, when run on an electronic device, causes the electronic device to perform a method of implementing the above aspects of embodiments of the present application and any possible involvement of the aspects.
In addition, for technical effects brought by any one implementation manner of the second aspect to the fifth aspect, reference may be made to technical effects brought by different implementation manners of the first aspect, and details are not described here.
The face recognition method and the intelligent device provided by the invention have the following beneficial effects:
according to the face recognition method and the intelligent device, the face image obtained through recognition is converted to generate the corresponding cartoon type face image, so that no matter face information is registered or face recognition is carried out, the real face information of the user cannot be stored, the personal information of the user can be effectively protected, and the fact that some mechanisms capable of breaking face recognition acquire the real face image or the real face characteristic information of the user is avoided.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic structural diagram of an intelligent device according to an embodiment of the present invention;
fig. 2 is a schematic view of a face image obtained by performing face recognition on a reference image through a face image recognition model according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a face image obtained after segmentation according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a training method for generating a confrontation network model according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a training method for generating a confrontation network model according to an embodiment of the present invention;
fig. 6 is a schematic diagram illustrating a cartoon image covering a real face image according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a geometric normalization processing method for a target image according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a cartoon image for correcting and subsequently generating a face image according to an embodiment of the present invention;
fig. 9 is a schematic diagram of a method for training a face feature extraction model according to an embodiment of the present invention;
fig. 10 is a schematic diagram of a face recognition method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The term "and/or" in the embodiments of the present invention describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The application scenario described in the embodiment of the present invention is for more clearly illustrating the technical solution of the embodiment of the present invention, and does not form a limitation on the technical solution provided in the embodiment of the present invention, and it can be known by a person skilled in the art that with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present invention is also applicable to similar technical problems. In the description of the present invention, the term "plurality" means two or more unless otherwise specified.
The face recognition technology is a series of related technologies that a camera or a camera is used for collecting images or video streams containing faces, the faces are automatically detected and tracked in the images, and then the faces are recognized, although the face recognition technology can facilitate our lives, due to the existence of the phenomena that an anti-neural network attacks and a face recognition mechanism is broken, even if face characteristic information is stored in face recognition instead of a face picture, the anti-neural network is generated to restore the face characteristic information to an original picture, the face information of a user has uniqueness, once the face information is leaked, for example, some important mechanisms such as banks can use faces to verify, and when the face information of the user is leaked, the property of the user can be greatly lost.
On the other hand, most of the existing intelligent algorithms of intelligent household appliances need to acquire video information, and the acquired video information mostly comprises face information no matter the algorithms are applied to face identity verification, gesture behavior recognition and the like. Video data in a family belongs to privacy data, while video information with a human face is more sensitive information, and once exposure and infringement occur, personal privacy also brings great risk.
In view of the above, an embodiment of the present application provides a smart device, as shown in fig. 1, which may include:
a display 101 configured to display a screen;
an image collector 102 configured to: collecting a reference image containing face information of a user;
a processor 103 configured to:
carrying out face recognition on a reference image collected by an image collector to obtain a face image; generating a cartoon type target image based on the face image; extracting face feature information of a target image, and comparing the face feature information with at least one piece of face feature information stored in advance; and verifying the reference image according to the comparison result.
Based on the intelligent equipment, the face image obtained by recognition is converted to generate a corresponding cartoon type face image, so that the real face information of the user cannot be stored no matter the face information is registered or the face is recognized, when other recognition algorithms need to collect gesture information and the like, the collected real face image can be converted into the cartoon image, the personal information of the user can be effectively protected, and the situation that some mechanisms capable of breaking the face recognition acquire the real face image or the real face feature information of the user is avoided.
Optionally, in this embodiment of the application, the display may be used to display a picture, when the intelligent terminal performs face recognition, gesture recognition or other operations on the user, the intelligent terminal may but is not limited to display a cartoon-type target image generated based on a face image obtained through recognition on the display, or after a reference image is verified, display a prompt message indicating that verification is successful or unsuccessful on the display, specifically, the prompt message indicating that verification is successful may be displayed on the display in a text box form, and when verification is successful, the prompt message indicating that verification is unsuccessful may be displayed on the display, and when a face is not recognized for a long time, the prompt message indicating that a face is not recognized may be displayed on the display, and details are not repeated herein.
The intelligent device in the embodiment of the application may be an intelligent device having a face recognition function, a gesture recognition function, or other functions that may acquire a face image, and optionally, may be but not limited to a mobile terminal or an intelligent home, and the like, which is not limited herein.
In addition, the image collector may be an image collecting structure such as a camera, as long as the image collection can be realized, and the specific implementation structure is not specifically limited herein, in the embodiments of the present application, the image collector can collect a section of video stream including the face information of the user in real time when receiving the image collecting command, and specifically, can be set to collect the video stream for a preset time length, or stopping the acquisition when the acquired video stream reaches the preset size or the number of the acquired video stream including the face information images of the user reaches the preset requirement, wherein each frame of image in the video stream can be used as a reference image collected by the image collector, as another optional implementation, and when an image acquisition instruction is received, acquiring a plurality of images comprising the face information of the user as the reference images within a preset time interval.
Optionally, the image collector sends the collected reference image to the processor, the processor receives the reference image sent by the image collector, and the reference image may be a reference image collected by the image collector in real time through an image collecting structure such as a camera.
Optionally, the processor is configured to perform face recognition on a reference image acquired by the image acquirer through the face recognition model to obtain a face image, specifically, position information of a user face and a corresponding confidence level may be obtained through the face recognition model, and since the reference image possibly acquired by the image acquirer does not include the user face information or a non-face image may be recognized in a recognition process, a confidence level is set in the embodiment of the present application to determine whether the recognition result is accurate, it may be defined that the face image is recognized when the confidence level is greater than a preset threshold value, specifically, the preset threshold value may be set to 0.9 in the embodiment of the present application, where the preset threshold value is a configurable parameter, and a person skilled in the art may set the preset threshold value according to actual needs, which is not limited herein.
Fig. 2 shows an example of obtaining a face image by performing face recognition on a reference image acquired by an image acquisition device through a face image recognition model, where a square frame in fig. 2 represents a recognized face image, specifically, position information of the face image can be determined through coordinate information of at least two vertexes in the square frame, a confidence of the image in the square frame is labeled at an upper left corner outside the square frame, and a confidence corresponding to the face image in the square frame in fig. 2 is 0.965.
Alternatively, after the face image is recognized by the face recognition model and the coordinate position of the face image is determined, the face image is segmented from the reference image based on the coordinate position, as shown in fig. 3, which is an example of the segmented face image.
Optionally, because the image collected by the image collector of the intelligent device may have a condition of low data pixel, the low-pixel face image may lose more detail information when being converted into the cartoon face image, so that the recognition accuracy of the cartoon face image is low. Therefore, after the face image is obtained, the pixel value of the face part can be increased by but not limited to using a super-resolution reconstruction technology, and then the high-resolution face image based on the super-resolution reconstruction is converted into a cartoon image, so that more details can be reserved. The super-resolution reconstruction technique of images refers to converting a given low-resolution image into a corresponding high-resolution image through a specific algorithm. Specifically, firstly, a bicubic interpolation is adopted to amplify the low-resolution image into a target size, then nonlinear mapping is fitted through a three-layer convolution network, and finally, a high-resolution image is output.
Optionally, the processor is further configured to:
converting the face image into a target image of a cartoon type based on the trained generated confrontation network model;
and extracting the face feature information of the target image based on the trained face feature extraction model.
Optionally, the embodiment of the application may be, but is not limited to, generating a confrontation network model to generate a cartoon-type target image from a face image, and of course, other image generation models may also be used, which is not limited herein;
specifically, the generation of the countermeasure network model needs pre-training, and after the recognized face image is input into the generated countermeasure network model obtained after training, the face image is converted into a target image of a cartoon type and output, optionally, the generation of the countermeasure model is generated in a training manner as shown in fig. 4, and includes:
step S401, a first sample set comprising a plurality of original face images and corresponding original cartoon images is obtained, and multiple times of training are carried out on an initially generated confrontation network model based on the first sample set to obtain a trained generated confrontation network model, wherein each training process comprises the following steps;
optionally, the initially generated confrontation network model is trained for multiple times based on the first sample set, and when a training end condition is met, the training is ended to obtain the trained generated confrontation network model, where the training end condition in the embodiment of the present application may be that the training is ended when the training frequency reaches a preset requirement, or the training is ended when a loss value of at least one loss function corresponding to the training process is determined to meet a set condition, and the generated confrontation network model is obtained after the training is ended.
Step S402, selecting a first sample from a first sample set;
optionally, each time training is performed, the original face image and the corresponding original cartoon image are selected from the first sample set.
Step S403, inputting a first sample into the initially generated confrontation network model, determining a first loss value based on the cartoon image output by the initially generated confrontation network model and the corresponding original cartoon image, and performing parameter adjustment on the initially generated confrontation network model according to the first loss value;
optionally, the initially generated confrontation network model includes a plurality of generators and a plurality of discriminators, where the generators are configured to generate images, and the discriminators are configured to discriminate whether the images generated by the generators are accurate, in this embodiment of the application, after the original face image and the corresponding original cartoon image are input to the initially generated confrontation network model, the corresponding cartoon image is generated by the generators and serves as the cartoon image output by the initially generated confrontation network model;
optionally, the first loss value determined based on the cartoon image output by the initially generated network model and the corresponding original cartoon image in the embodiment of the present application includes the following two aspects:
1) penalty function (adescarial Loss): the method is used for representing the difference between the generated cartoon image and the corresponding original cartoon image, and carrying out parameter adjustment on an initially generated countermeasure network model through a countermeasure loss value, wherein the parameter adjustment is mainly carried out on the generator, so that the generated cartoon image is closer to the original cartoon image;
2) total Variation Loss function (Total Variation Loss): the method is used for representing the difference of image edge information between the generated cartoon image and the corresponding original cartoon image, and performing parameter adjustment on the initially generated countermeasure network model through a total variation loss value, wherein the parameter adjustment is mainly performed on the generator, so that the generated cartoon image is closer to the original cartoon image.
And after a first loss value is obtained through the loss function, parameter adjustment is carried out on the initially generated confrontation network model according to the first loss value.
Step S404, inputting the output cartoon image and the corresponding original face image into an image generation model, determining a second loss value based on the real face image and the original face image output by the image generation model, and performing parameter adjustment on the image generation model according to the second loss value;
optionally, the image generation model may include, but is not limited to, a model for generating an antagonistic network, and of course, other neural network models capable of generating an image are also applicable to the present application, and are not limited herein;
optionally, the image generation network model and the initially generated confrontation network model may be two neural network models, and correspondingly, if the image generation network model is the generated confrontation network model, the image generation network model also includes a plurality of generators and a plurality of discriminators.
In the embodiment of the application, after the output cartoon image and the corresponding original face image are input to generate the confrontation network model, the corresponding real face image is generated through the generator and is used as the real face image output by the generated confrontation network model;
similarly, the second loss value determined based on the real face image and the corresponding original face image output by the generation confrontation network model in the embodiment of the present application includes the following two aspects:
1) penalty function (adescarial Loss): the system comprises a generator, a parameter adjustment module and a parameter adjustment module, wherein the generator is mainly used for adjusting the parameters of the generator so as to enable the generated real face image to be closer to an original face image;
2) total Variation Loss function (Total Variation Loss): the method is used for representing the image edge information difference between the generated real face image and the corresponding original face image, and performing parameter adjustment on the generated confrontation network model through a total variation loss value, wherein the parameter adjustment is mainly performed on the generator, so that the generated real face image is closer to the original face image.
And after a second loss value is obtained through the loss function, parameter adjustment is carried out on the generated confrontation network model according to the second loss value.
Step S405, determining a third loss value according to the original face image and the real face image, and performing parameter adjustment on the initially generated confrontation network model according to the third loss value.
Optionally, the third loss function is used to set different loss weights for a specified target region of the face image, the third loss function may be, but is not limited to, an attention cycle loss function, the specified target region of the face image may be, but is not limited to, an eye shadow, a pupil, a nasal wing, a mouth corner, and the like, and by the third loss function, the initial neural network model may be guided to generate better main details of the face, more attention is given to the specified target region, and more details are reserved to be beneficial to the recognition and verification of a subsequent cartoon face;
specifically, the third loss value is determined according to the real face image and the original face image output by the image generation model, and then the initially generated confrontation network model is subjected to parameter adjustment according to the third loss value, as shown in fig. 5, which is a training process of the generated confrontation network model.
And converting the face image into a cartoon type target image based on the trained generated confrontation network model, and then optionally, not only limiting the target image to the cartoon type image, but also being applicable to other modes which can protect privacy and can perform face recognition.
Optionally, the obtained target image of the cartoon type is covered with the recognized face image, as shown in fig. 6;
optionally, in order to improve the stability and accuracy of face recognition, geometric normalization processing needs to be performed on the obtained target image of the cartoon type, where geometric normalization refers to converting a face in an image to a certain position according to a face positioning result and converting the face into a specified size, and is used to enable the face image or face feature information stored during registration to be located at the same position as or have the same size as the acquired face image or face feature information during face recognition.
The step of performing geometric normalization processing on the target image is shown in fig. 7, and includes:
step S701, obtaining an affine transformation matrix based on the face key point information of the face image and the face key point information of a preset standard face image;
optionally, in this embodiment of the application, the face key point information of the face image is determined based on a face key point detection model, where the key point detection model includes, but is not limited to, a face key point detection model obtained based on deep convolutional neural network training, and detects face key points of the face, and optionally, the preset face key points include, but are not limited to, eyeball centers of eyes on both sides, a nose tip, and key points of mouth corners on both sides.
In step S702, the face image is corrected based on the affine transformation matrix.
As shown in fig. 8, a process of correcting a face image and subsequently generating a cartoon image is provided for the embodiment of the present application.
Optionally, the embodiment of the application may be, but is not limited to, extracting the face feature information of the target image by using a face feature extraction model, where the face feature extraction model is a cartoon type face feature extraction model;
specifically, the cartoon-type face feature extraction model needs to be pre-trained, and after the target image of the cartoon type is input into the trained face feature extraction model, corresponding face feature information is output, optionally, the face feature extraction model is generated by a training mode as shown in fig. 9, including:
step S901, a second sample set comprising a plurality of cartoon images is obtained, and the initial face feature extraction model is trained for a plurality of times based on the second sample set to obtain a trained face feature extraction model, wherein each training process comprises the following steps;
optionally, training the face feature extraction model for multiple times based on the second sample set, and when a training end condition is met, ending the training to obtain the trained face feature extraction model, where the training end condition in the embodiment of the present application may be that the training is ended when the training frequency reaches a preset requirement, or the training is ended when a loss value of at least one loss function corresponding to the training process is determined to meet a set condition, and the face feature extraction model is obtained after the training is ended;
specifically, the embodiment of the present application sets that the loss function corresponding to the face feature extraction model takes zero, and ends the training, and optionally, may end the training when the loss function corresponding to the face feature model takes zero for multiple times.
Step S902, selecting at least one cartoon image of the same user from a second sample set, and extracting the face feature information of the at least one cartoon image of the same user based on an initial face feature extraction model;
optionally, the initial face feature extraction model of the application is a convolutional neural network model capable of extracting real face feature information, the initial face feature extraction model needs to be trained through a cartoon image to obtain the cartoon face feature extraction model, namely, the face feature extraction model corresponding to the application, when a face is registered, the obtained face information is stored in a registration database, wherein the face feature information stored in the registration database is a face feature vector, and a processor can determine whether different feature vectors correspond to one person or not by calculating similarities among different feature vectors, so that the application has privacy security by extracting the feature information of the cartoon face;
optionally, in the embodiment of the application, two different cartoon images of the same user are selected to extract face feature information based on the initial face feature extraction model;
step S903, selecting at least one cartoon image of another user from the second sample set, and extracting the face feature information of the at least one cartoon image of the another user based on the initial face feature extraction model;
optionally, in the embodiment of the present application, a cartoon image of another user is selected to extract the facial feature information based on the initial facial feature extraction model.
Step S904, based on the face feature information of at least one cartoon image of the same user extracted by the initial face feature extraction model and the face feature information of at least one cartoon image of another user, determining a fourth loss value;
optionally, the loss function corresponding to the embodiment of the present application is as follows:
L(A，P，N)＝max(‖f(A)-f(P)‖2-‖f(A)-f(N)‖2+α,0)
in the embodiment of the application, the extracted feature values are one-dimensional 64 feature values, because the cartoon images can retain human face detail features as much as possible, but can also lose many detail features at the same time;
and a is a hyper-parameter used for indicating that the distance between different human faces is at least a distance, and a is a configurable parameter.
And step S905, performing parameter adjustment on the initial human face feature extraction model according to the fourth loss value.
Optionally, the extracted face feature information is compared with at least one piece of face feature information stored in advance, specifically, the similarity between the face feature information of the target image and the at least one piece of face feature information stored in advance is calculated, and the reference image is verified based on the similarity;
optionally, in the embodiment of the present application, a formula for calculating similarity between face feature information is as follows:
the Similarity is Similarity, A is a face feature vector of a cartoon type target image obtained based on a reference image, B is at least one face feature vector stored in advance, and Ai and Bi are components of the face feature vectors A and B respectively.
Optionally, after the reference image is obtained, the face feature information of the reference image corresponding to the cartoon image is obtained based on the above process and may be stored in a registration database, as another optional implementation manner, when performing identification verification, the face feature information of the reference image corresponding to the cartoon image is obtained based on the above process and is compared with the face feature information in the registration database one by one, and when it is determined that the similarity is greater than the preset threshold, the verification is successful.
Based on the same inventive concept, an embodiment of the present invention provides a face recognition method, which is implemented by using the above-mentioned intelligent device provided in the embodiment of the present invention, and as shown in fig. 10, the method may include:
step S1001, collecting a reference image containing user face information to perform face recognition to obtain a face image;
step S1002, generating a cartoon type target image based on the face image;
step S1003, extracting the face feature information of the target image, and comparing the face feature information with at least one face feature information stored in advance;
step S1004, verifying the reference image according to the comparison result.
Optionally, generating a cartoon-type target image based on the face image includes:
converting the face image into a target image of a cartoon type based on the trained generated confrontation network model;
extracting the face feature information of the target image, including:
and extracting the face feature information of the target image based on the trained face feature extraction model.
Optionally, the generating an antagonistic network model is generated by the following training mode:
the method comprises the steps of obtaining a first sample set comprising a plurality of original face images and corresponding original cartoon images, training an initially generated confrontation network model for a plurality of times based on the first sample set, and obtaining a trained generated confrontation network model, wherein each training process comprises the following steps:
selecting a first sample from the first sample set;
inputting the first sample into an initially generated confrontation network model, determining a first loss value based on a cartoon image output by the initially generated confrontation network model and a corresponding original cartoon image, and performing parameter adjustment on the initially generated confrontation network model according to the first loss value;
inputting the output cartoon image and the corresponding original face image into an image generation model, determining a second loss value based on the real face image and the original face image output by the image generation model, and performing parameter adjustment on the image generation model according to the second loss value;
and determining a third loss value according to the original face image and the real face image, and performing parameter adjustment on the initially generated confrontation network model according to the third loss value.
Optionally, the face feature extraction model is generated by the following training method:
acquiring a second sample set comprising a plurality of cartoon images, and training the initial human face feature extraction model for a plurality of times based on the second sample set to obtain a trained human face feature extraction model, wherein each training process comprises the following steps:
selecting at least one cartoon image of the same user from the second sample set, and extracting face feature information of the at least one cartoon image of the same user based on an initial face feature extraction model;
selecting at least one cartoon image of another user from the second sample set, and extracting face feature information of the at least one cartoon image of the another user based on an initial face feature extraction model;
determining a fourth loss value based on the face feature information of the at least one cartoon image of the same user extracted by the initial face feature extraction model and the face feature information of the at least one cartoon image of another user;
and adjusting parameters of the initial face feature extraction model according to the fourth loss value.
Optionally, before generating the cartoon-type target image based on the face image, the method includes:
obtaining an affine transformation matrix based on the face key point information of the face image and the face key point information of a preset standard face image;
and correcting the face image based on the affine transformation matrix.
For implementation of the above steps, reference may be made to the specific embodiment of the foregoing intelligent device, and repeated details are not repeated.
An embodiment of the present invention further provides a computer-readable storage medium, which includes instructions, and when the computer-readable storage medium runs on a computer, the computer is caused to execute the method for face recognition provided in the foregoing embodiment.
An embodiment of the present application further provides a computer program product, which includes a computer program, where the computer program includes program instructions, and when the program instructions are executed by an electronic device, the electronic device is caused to execute the method for face recognition provided in the foregoing embodiment.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The technical solutions provided by the present application are introduced in detail, and the present application applies specific examples to explain the principles and embodiments of the present application, and the descriptions of the above examples are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A smart device, comprising:
a display configured to display a screen;
an image collector configured to: collecting a reference image containing face information of a user;
a processor configured to:
carrying out face recognition on the reference image collected by the image collector to obtain a face image; generating a cartoon type target image based on the face image; extracting the face feature information of the target image, and comparing the face feature information with at least one face feature information stored in advance; and verifying the reference image according to the comparison result.
2. The smart device of claim 1, wherein the processor is configured to:
converting the face image into a target image of a cartoon type based on the trained generated confrontation network model;
and extracting the face feature information of the target image based on the trained face feature extraction model.
3. The smart device of claim 2, wherein the processor is configured to
The method comprises the steps of obtaining a first sample set comprising a plurality of original face images and corresponding original cartoon images, training an initially generated confrontation network model for a plurality of times based on the first sample set, and obtaining a trained generated confrontation network model, wherein each training process comprises the following steps:
selecting a first sample from the first sample set;
inputting the first sample into an initially generated confrontation network model, determining a first loss value based on a cartoon image output by the initially generated confrontation network model and a corresponding original cartoon image, and performing parameter adjustment on the initially generated confrontation network model according to the first loss value;
inputting the output cartoon image and the corresponding original face image into an image generation model, determining a second loss value based on the real face image and the original face image output by the image generation model, and performing parameter adjustment on the image generation model according to the second loss value;
and determining a third loss value according to the original face image and the real face image, and performing parameter adjustment on the initially generated confrontation network model according to the third loss value.
4. The smart device of claim 2, wherein the processor is configured to:
acquiring a second sample set comprising a plurality of cartoon images, and training the initial human face feature extraction model for a plurality of times based on the second sample set to obtain a trained human face feature extraction model, wherein each training process comprises the following steps:
selecting at least one cartoon image of the same user from the second sample set, and extracting face feature information of the at least one cartoon image of the same user based on an initial face feature extraction model;
selecting at least one cartoon image of another user from the second sample set, and extracting face feature information of the at least one cartoon image of the another user based on an initial face feature extraction model;
determining a fourth loss value based on the face feature information of the at least one cartoon image of the same user extracted by the initial face feature extraction model and the face feature information of the at least one cartoon image of another user;
and adjusting parameters of the initial face feature extraction model according to the fourth loss value.
5. The smart device of claim 1, wherein the processor is configured to:
obtaining an affine transformation matrix based on the face key point information of the face image and the face key point information of a preset standard face image;
and correcting the face image based on the affine transformation matrix.
6. A face recognition method, comprising:
acquiring a reference image containing user face information to perform face recognition to obtain a face image;
generating a cartoon type target image based on the face image;
extracting the face feature information of the target image, and comparing the face feature information with at least one face feature information stored in advance;
and verifying the reference image according to the comparison result.
7. The method of claim 6, wherein generating a cartoon-type target image based on the facial image comprises:
converting the face image into a target image of a cartoon type based on the trained generated confrontation network model;
extracting the face feature information of the target image, including:
and extracting the face feature information of the target image based on the trained face feature extraction model.
8. The method of claim 7, wherein the generating the countermeasure network model is generated by training:
the method comprises the steps of obtaining a first sample set comprising a plurality of original face images and corresponding original cartoon images, training an initially generated confrontation network model for a plurality of times based on the first sample set, and obtaining a trained generated confrontation network model, wherein each training process comprises the following steps:
selecting a first sample from the first sample set;
inputting the first sample into an initially generated confrontation network model, determining a first loss value based on a cartoon image output by the initially generated confrontation network model and a corresponding original cartoon image, and performing parameter adjustment on the initially generated confrontation network model according to the first loss value;
inputting the output cartoon image and the corresponding original face image into an image generation model, determining a second loss value based on the real face image and the original face image output by the image generation model, and performing parameter adjustment on the image generation model according to the second loss value;
and determining a third loss value according to the original face image and the real face image, and performing parameter adjustment on the initially generated confrontation network model according to the third loss value.
9. The method of claim 7, wherein the face feature extraction model is generated by training:
acquiring a second sample set comprising a plurality of cartoon images, and training the initial human face feature extraction model for a plurality of times based on the second sample set to obtain a trained human face feature extraction model, wherein each training process comprises the following steps:
selecting at least one cartoon image of the same user from the second sample set, and extracting face feature information of the at least one cartoon image of the same user based on an initial face feature extraction model;
selecting at least one cartoon image of another user from the second sample set, and extracting face feature information of the at least one cartoon image of the another user based on an initial face feature extraction model;
determining a fourth loss value based on the face feature information of the at least one cartoon image of the same user extracted by the initial face feature extraction model and the face feature information of the at least one cartoon image of another user;
and adjusting parameters of the initial face feature extraction model according to the fourth loss value.
10. The method of claim 6, wherein generating the cartoon-type target image based on the face image comprises:
obtaining an affine transformation matrix based on the face key point information of the face image and the face key point information of a preset standard face image;
and correcting the face image based on the affine transformation matrix.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010460440.0A CN113486688A (en) | 2020-05-27 | 2020-05-27 | Face recognition method and intelligent device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010460440.0A CN113486688A (en) | 2020-05-27 | 2020-05-27 | Face recognition method and intelligent device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113486688A true CN113486688A (en) | 2021-10-08 |
Family
ID=77932554
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010460440.0A Pending CN113486688A (en) | 2020-05-27 | 2020-05-27 | Face recognition method and intelligent device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113486688A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023160448A1 (en) * | 2022-02-24 | 2023-08-31 | 北京字跳网络技术有限公司 | Image processing method and apparatus, and device and storage medium |
| CN117894083A (en) * | 2024-03-14 | 2024-04-16 | 中电科大数据研究院有限公司 | Image recognition method and system based on deep learning |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107577985A (en) * | 2017-07-18 | 2018-01-12 | 南京邮电大学 | The implementation method of the face head portrait cartooning of confrontation network is generated based on circulation |
| CN108564127A (en) * | 2018-04-19 | 2018-09-21 | 腾讯科技（深圳）有限公司 | Image conversion method, device, computer equipment and storage medium |
| CN109117801A (en) * | 2018-08-20 | 2019-01-01 | 深圳壹账通智能科技有限公司 | Method, apparatus, terminal and the computer readable storage medium of recognition of face |
| CN109376582A (en) * | 2018-09-04 | 2019-02-22 | 电子科技大学 | A kind of interactive human face cartoon method based on generation confrontation network |
| CN109635745A (en) * | 2018-12-13 | 2019-04-16 | 广东工业大学 | A method of Multi-angle human face image is generated based on confrontation network model is generated |
| CN109993072A (en) * | 2019-03-14 | 2019-07-09 | 中山大学 | The low resolution pedestrian weight identifying system and method generated based on super resolution image |
| CN110659582A (en) * | 2019-08-29 | 2020-01-07 | 深圳云天励飞技术有限公司 | Image conversion model training method, heterogeneous face recognition method, device and equipment |
| CN110717977A (en) * | 2019-10-23 | 2020-01-21 | 网易（杭州）网络有限公司 | Method and device for processing face of game character, computer equipment and storage medium |
| CN110765843A (en) * | 2019-09-03 | 2020-02-07 | 平安科技（深圳）有限公司 | Face verification method and device, computer equipment and storage medium |
| WO2020063744A1 (en) * | 2018-09-30 | 2020-04-02 | 腾讯科技（深圳）有限公司 | Face detection method and device, service processing method, terminal device, and storage medium |
| CN111191564A (en) * | 2019-12-26 | 2020-05-22 | 三盟科技股份有限公司 | Multi-pose face emotion recognition method and system based on multi-angle neural network |
-
2020
- 2020-05-27 CN CN202010460440.0A patent/CN113486688A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107577985A (en) * | 2017-07-18 | 2018-01-12 | 南京邮电大学 | The implementation method of the face head portrait cartooning of confrontation network is generated based on circulation |
| CN108564127A (en) * | 2018-04-19 | 2018-09-21 | 腾讯科技（深圳）有限公司 | Image conversion method, device, computer equipment and storage medium |
| CN109117801A (en) * | 2018-08-20 | 2019-01-01 | 深圳壹账通智能科技有限公司 | Method, apparatus, terminal and the computer readable storage medium of recognition of face |
| CN109376582A (en) * | 2018-09-04 | 2019-02-22 | 电子科技大学 | A kind of interactive human face cartoon method based on generation confrontation network |
| WO2020063744A1 (en) * | 2018-09-30 | 2020-04-02 | 腾讯科技（深圳）有限公司 | Face detection method and device, service processing method, terminal device, and storage medium |
| CN109635745A (en) * | 2018-12-13 | 2019-04-16 | 广东工业大学 | A method of Multi-angle human face image is generated based on confrontation network model is generated |
| CN109993072A (en) * | 2019-03-14 | 2019-07-09 | 中山大学 | The low resolution pedestrian weight identifying system and method generated based on super resolution image |
| CN110659582A (en) * | 2019-08-29 | 2020-01-07 | 深圳云天励飞技术有限公司 | Image conversion model training method, heterogeneous face recognition method, device and equipment |
| CN110765843A (en) * | 2019-09-03 | 2020-02-07 | 平安科技（深圳）有限公司 | Face verification method and device, computer equipment and storage medium |
| CN110717977A (en) * | 2019-10-23 | 2020-01-21 | 网易（杭州）网络有限公司 | Method and device for processing face of game character, computer equipment and storage medium |
| CN111191564A (en) * | 2019-12-26 | 2020-05-22 | 三盟科技股份有限公司 | Multi-pose face emotion recognition method and system based on multi-angle neural network |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023160448A1 (en) * | 2022-02-24 | 2023-08-31 | 北京字跳网络技术有限公司 | Image processing method and apparatus, and device and storage medium |
| CN117894083A (en) * | 2024-03-14 | 2024-04-16 | 中电科大数据研究院有限公司 | Image recognition method and system based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10810423B2 (en) | Iris liveness detection for mobile devices | |
| CN107133608A (en) | Identity authorization system based on In vivo detection and face verification | |
| TWI335544B (en) | Iris recognition system | |
| US20230087657A1 (en) | Assessing face image quality for application of facial recognition | |
| CN112364827B (en) | Face recognition method, device, computer equipment and storage medium | |
| US9292752B2 (en) | Image processing device and image processing method | |
| CN111339897B (en) | Living body identification method, living body identification device, computer device, and storage medium | |
| CN110705353A (en) | Method and device for identifying face to be shielded based on attention mechanism | |
| CN111274947A (en) | Multi-task multi-thread face recognition method, system and storage medium | |
| CN113642639B (en) | Living body detection method, living body detection device, living body detection equipment and storage medium | |
| Haji et al. | Real time face recognition system (RTFRS) | |
| CN112560683A (en) | Method and device for identifying copied image, computer equipment and storage medium | |
| CN113011544B (en) | Face biological information identification method, system, terminal and medium based on two-dimensional code | |
| CN113486688A (en) | Face recognition method and intelligent device | |
| CN112101296B (en) | Face registration method, face verification method, device and system | |
| CN113033243A (en) | Face recognition method, device and equipment | |
| CN112101479B (en) | Hair style identification method and device | |
| WO2023028947A1 (en) | Palm vein non-contact three-dimensional modeling method and apparatus, and authentication method | |
| CN112800941B (en) | Face anti-fraud method and system based on asymmetric auxiliary information embedded network | |
| Li et al. | Video image moving target recognition method based on generated countermeasure network | |
| CN109242489B (en) | Authentication mode selection method and device | |
| CN112380966A (en) | Monocular iris matching method based on feature point reprojection | |
| Sun et al. | Presentation attacks in palmprint recognition systems | |
| CN116978130A (en) | Image processing method, image processing device, computer device, storage medium, and program product | |
| CN117671751A (en) | Face recognition method, system and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211008 |