CN110136054B - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN110136054B CN110136054B CN201910409754.5A CN201910409754A CN110136054B CN 110136054 B CN110136054 B CN 110136054B CN 201910409754 A CN201910409754 A CN 201910409754A CN 110136054 B CN110136054 B CN 110136054B
- Authority
- CN
- China
- Prior art keywords
- image
- eye
- processed
- eye image
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims description 20
- 238000012545 processing Methods 0.000 claims abstract description 152
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000012549 training Methods 0.000 claims description 48
- 210000000744 eyelid Anatomy 0.000 claims description 21
- 241000255789 Bombyx mori Species 0.000 claims description 19
- 238000004891 communication Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 8
- 238000012805 post-processing Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000013145 classification model Methods 0.000 description 3
- 239000002537 cosmetic Substances 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 206010003591 Ataxia Diseases 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 206010010947 Coordination abnormal Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 208000016290 incoordination Diseases 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G06T3/04—
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Abstract
The embodiment of the disclosure discloses a method and a device for processing an image. One embodiment of the method comprises the following steps: determining an eye image to be processed from the acquired face image to be processed; inputting an eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed; and replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing. The embodiment realizes the targeted addition or removal of the preset eye objects to the face image to be processed.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to an image processing method and device.
Background
Various cosmetic Applications (APP) generally involve making up face images uploaded by users, such as adding double eyelid or lying silkworm to the face images.
Currently, in order to add double eyelid or lying silkworm in a face image, the double eyelid or lying silkworm is mainly implemented through preset mapping. Specifically, the position of the eye part in the face image is determined, and then the preset map is placed at the corresponding position in the face image according to the position of the eye part.
Disclosure of Invention
The embodiment of the disclosure provides an image processing method and device.
In a first aspect, embodiments of the present disclosure provide an image processing method, the method including: determining an eye image to be processed from the acquired face image to be processed; inputting an eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed; and replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing.
In some embodiments, the predetermined ocular object comprises at least one of: double eyelid, lying silkworm.
In some embodiments, the image processing model is obtained by training a sample set, where samples in the sample set are image pairs including a sample first image and a sample second image, a preset eye object does not exist in an area corresponding to the preset eye object in the sample first image, and a preset eye object exists in an area corresponding to the preset eye object in the sample second image.
In some embodiments, the image processing model is trained by: taking a first sample image included in a sample in the sample set as an input of an initial model, taking a second sample image corresponding to the input first sample image as an expected output of the initial model, and training to obtain an image processing model.
In some embodiments, the image processing model is trained by: taking a sample second image included in a sample in the sample set as an input of an initial model, taking a sample first image corresponding to the input sample second image as an expected output of the initial model, and training to obtain an image processing model.
In some embodiments, the samples in the sample set are obtained by: cutting out an eye image from a first face image in the acquired first face image set to obtain a first eye image set; performing horizontal overturning on the first eye images in the first eye image set to obtain a second eye image set; selecting an eye image, in which a preset eye object does not exist, from the first eye image set and the second eye image set in a region corresponding to the preset eye object, so as to obtain a third eye image set; inputting a third eye image in the third eye image set into a pre-trained sample processing model to obtain a fourth eye image set, wherein the sample processing model is used for adding a preset eye object to the third eye image; samples in the sample set are generated based on a fourth eye image selected from the fourth eye image set and a corresponding third eye image selected from the third eye image set.
In some embodiments, determining the eye image to be processed from the obtained face image to be processed includes: based on the key points extracted from the acquired face image to be processed, the eye image to be processed with the preset size is cut from the face image to be processed.
In some embodiments, the above method further comprises: and sending the processed face image to the terminal equipment in communication connection so that the terminal equipment displays the processed face image.
In a second aspect, embodiments of the present disclosure provide an image processing apparatus including: the determining unit is configured to determine an eye image to be processed from the acquired face image to be processed; the processing unit is configured to input an eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed; and the generating unit is configured to replace the eye image to be processed in the face image to be processed with the eye image after processing so as to generate the face image after processing.
In some embodiments, the predetermined ocular object comprises at least one of: double eyelid, lying silkworm.
In some embodiments, the image processing model is obtained by training a sample set, where samples in the sample set are image pairs including a sample first image and a sample second image, a preset eye object does not exist in an area corresponding to the preset eye object in the sample first image, and a preset eye object exists in an area corresponding to the preset eye object in the sample second image.
In some embodiments, the image processing model is trained by: taking a first sample image included in a sample in the sample set as an input of an initial model, taking a second sample image corresponding to the input first sample image as an expected output of the initial model, and training to obtain an image processing model.
In some embodiments, the image processing model is trained by: taking a sample second image included in a sample in the sample set as an input of an initial model, taking a sample first image corresponding to the input sample second image as an expected output of the initial model, and training to obtain an image processing model.
In some embodiments, the samples in the sample set are obtained by: cutting out an eye image from a first face image in the acquired first face image set to obtain a first eye image set; performing horizontal overturning on the first eye images in the first eye image set to obtain a second eye image set; selecting an eye image, in which a preset eye object does not exist, from the first eye image set and the second eye image set in a region corresponding to the preset eye object, so as to obtain a third eye image set; inputting a third eye image in the third eye image set into a pre-trained sample processing model to obtain a fourth eye image set, wherein the sample processing model is used for adding a preset eye object to the third eye image; samples in the sample set are generated based on a fourth eye image selected from the fourth eye image set and a corresponding third eye image selected from the third eye image set.
In some embodiments, the above-mentioned determining unit is further configured to: based on the key points extracted from the acquired face image to be processed, the eye image to be processed with the preset size is cut from the face image to be processed.
In some embodiments, the apparatus further comprises: and the sending unit is configured to send the processed face image to the terminal equipment in communication connection so as to enable the terminal equipment to display the processed face image.
In a third aspect, embodiments of the present disclosure provide a server comprising: one or more processors; a storage device having one or more programs stored thereon; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method as described in any of the implementations of the first aspect.
The image processing method and device provided by the embodiment of the disclosure can acquire a face image to be processed; then, determining an eye image to be processed from the acquired face image to be processed; then, the determined eye image to be processed can be input into a pre-trained image processing model to obtain a processed eye image; further, the eye image to be processed in the face image to be processed can be replaced by the obtained eye image after processing, so that the face image after processing can be generated. Therefore, the method and the device realize the targeted addition or removal of the preset eye objects to the face image to be processed.
Drawings
Other features, objects and advantages of the present disclosure will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;
FIG. 2 is a flow chart of one embodiment of an image processing method according to the present disclosure;
FIG. 3 is a schematic illustration of one application scenario of an image processing method according to an embodiment of the present disclosure;
FIG. 4 is a flow chart of yet another embodiment of an image processing method according to the present disclosure;
fig. 5 is a schematic structural view of an embodiment of an image processing apparatus according to the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, without conflict, the embodiments of the present disclosure and features of the embodiments may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary architecture 100 to which the image processing method or image processing apparatus of the present disclosure may be applied.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, a network 103, and a server 104. The network 103 is the medium used to provide communication links between the terminal devices 101, 102 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The terminal devices 101, 102 interact with the server 104 through the network 103 to receive or send messages or the like. Various communication client applications, such as an image processing application, a cosmetic application, a browser application, etc., may be installed on the terminal devices 101, 102.
The terminal devices 101 and 102 may be hardware or software. When the terminal devices 101, 102 are hardware, they may be various electronic devices having a display screen and supporting image processing, including but not limited to smartphones, tablet computers, laptop and desktop computers, and the like. When the terminal devices 101, 102 are software, they may be installed in the above-listed electronic devices, which may be implemented as a plurality of software or software modules, or as a single software or software module. The present invention is not particularly limited herein.
The server 104 may be a server providing various services. As an example, the server 104 may be a background server of the cosmetic application installed on the terminal device 101, 102. The server 104 may acquire the face image to be processed transmitted from the terminal device 101, 102, and then process the face image to be processed, thereby transmitting the generated processed face image to the terminal device 101, 102.
The server 104 may be hardware or software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., a plurality of software or software modules for providing distributed services), or as a single software or software module. The present invention is not particularly limited herein.
It should be noted that, the image processing method provided by the embodiments of the present disclosure is generally performed by the server 104, and accordingly, the image processing apparatus is generally disposed in the server 104.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of one embodiment of an image processing method according to the present disclosure is shown. The image processing method includes the steps of:
step 201, determining an eye image to be processed from the acquired face image to be processed.
In this embodiment, the execution subject of the image processing method (such as the server 104 shown in fig. 1) may acquire the face image to be processed from a terminal device (such as the terminal devices 101, 102 shown in fig. 1) connected locally or in communication. Here, the face image to be processed is typically an image on which a face is displayed.
In this embodiment, after the face image to be processed is acquired, the execution body may determine the eye image to be processed from the face image to be processed. Here, the eye image to be processed is typically an image in which the eye portion of the person is displayed in the above-described face image to be processed. The determined eye image to be processed may be an image in which only the left or right eye of the person is displayed, or may be an image in which both the left and right eyes of the person are displayed.
As an example, the above-described execution subject may have features extracted for the average eye image stored therein in advance. The average eye image may be an image obtained by averaging pixel values of corresponding pixel points in a large number of eye images. Firstly, the execution body may slide in the face image to be processed using a preset sliding window, then, a similarity may be calculated between the features extracted for the region where the sliding window is located and the obtained features, and then, a region with the maximum similarity may be determined as the eye image to be processed.
In some optional implementations of this embodiment, the executing body may crop a pre-set size of the eye image to be processed from the acquired face image to be processed based on the key points extracted from the face image to be processed.
First, the execution body may extract key points from the acquired face image to be processed. In general, the extracted keypoints may include keypoints extracted for contours of eye parts displayed in the face image to be processed. Of course, extracting the key points may also include extracting the key points for other parts of the face (e.g., face contours, pupils) displayed in the face image to be processed.
Then, the executing body may determine the approximate position of the eye portion in the face image to be processed according to the extracted key points. Further, an image of a preset size including the eye portion may be cut out from the face image to be processed as an eye image to be processed. Here, the preset size may be set according to actual requirements, which is not described herein.
In the implementation modes, through the key points extracted for the eye parts displayed in the face image to be processed, on one hand, the positions of the eye parts can be accurately positioned, so that the eye image to be processed can be accurately cut, on the other hand, the sliding window can be prevented from sliding for many times, the features can be extracted for the region of the sliding window for many times, and the time for determining the eye image to be processed is shortened.
Step 202, inputting the eye image to be processed into a pre-trained image processing model to obtain a processed eye image.
In this embodiment, after determining the eye image to be processed, the executing body may input the eye image to be processed into a pre-trained image processing model to obtain the processed eye image. The image processing model may be used to process an area corresponding to a preset eye object in the eye image to be processed.
The region corresponding to the preset eye object may be a region for adding or removing the preset eye object in the eye image to be processed. Alternatively, the predetermined ocular object may include at least one of double eyelid and lying silkworm. Accordingly, the region corresponding to the preset eye object may be a region for adding or removing at least one of the double eyelid and the lying silkworm in the above-described eye image to be processed. In general, the region for adding double eyelid may be a region closer to the upper eyelid displayed in the eye image to be processed, and the region for adding lying silkworm may be a region closer to the lower eyelid displayed in the eye image to be processed.
The image processing model may be a correspondence table stored in advance in the execution subject by a technician. The correspondence table can be obtained by processing a large number of eye images collected in advance by a technician. Specifically, the technician may add a preset ocular object to the ocular image or remove a preset ocular object displayed in the ocular image. In practice, in the correspondence table, the eye images and the images obtained after the eye image processing are in one-to-one correspondence. It is understood that the image obtained after the eye image processing may be an image obtained after adding a preset eye object to the eye image, or an image obtained after removing a preset eye object displayed in the eye image.
As an example, the executing body may input the determined eye image to be processed into the correspondence table, perform similarity matching with the eye image in the correspondence table, and then may acquire an image corresponding to the eye image with the maximum similarity in the correspondence table as the processed eye image. It is understood that the post-processing eye image adds or removes the pre-set eye object compared to the eye image to be processed.
In some optional implementations of this embodiment, the image processing model may also be a machine learning model trained using a sample set. The samples in the sample set may be an image pair including a first image of the sample and a second image of the sample. The preset eye object does not exist in the area corresponding to the preset eye object in the first sample image, and the preset eye object exists in the area corresponding to the preset eye object in the second sample image.
In some optional implementations of this embodiment, on the basis of the sample set, a first image of a sample included in the sample set may be used as an input of an initial model, and a second image of the sample corresponding to the input first image of the sample may be used as an expected output of the initial model, so as to train to obtain an image processing model.
The initial model may be an countermeasure generation network (Generative Adversarial Nets, GAN) for processing an eye image, such as a Pix2Pix (Pix is an abbreviation for Pixel) model. In the initial model training process, the difference between the first image as the input sample and the second image as the expected output sample in the same sample is small except for the region corresponding to the preset eye object.
The training steps of the image processing model are specifically described in the following steps S1 to S5.
Step S1, inputting a first sample image included in a sample selected from a sample set into an initial model to obtain a processed eye image of the input first sample image.
And S2, calculating the difference degree between the obtained processed eye image and the input first image of the sample by using a preset loss function, and calculating the complexity of the initial model by using a regularization term.
The preset loss function may be at least one of the following loss functions selected according to actual requirements: a 0-1 loss function, an absolute loss function, a square loss function, an exponential loss function, a logarithmic loss function, a hinge loss function, and the like. The regularization term can be any one of the following norms selected according to actual requirements: l0 norms, L1 norms, L2 norms, trace norms, kernel norms, etc.
And step S3, adjusting the structural parameters of the initial model according to the calculated difference degree and the complexity of the model.
In practice, the structural parameters of the initial model may be adjusted using any one of the following algorithms: BP (Back propagation) algorithm, GD (Gradient Descent) algorithm, and the like.
In step S4, in response to reaching the preset training end condition, the execution body for training the image processing model may determine that the training of the initial model is completed, and determine the initial model after the training is completed as the image processing model.
The preset training ending condition may include at least one of the following: the training time exceeds the preset duration; the training times exceed the preset times; the calculated degree of difference is less than a preset difference threshold.
And step S5, in response to the preset training ending condition not being met, the execution subject for training the image processing model can select unselected samples from the sample set, and the training step is continuously executed by using the adjusted initial model as the initial model.
The execution subject for training the image processing model may be the same as or different from the execution subject for image processing. If the two are the same, the execution subject for training the image processing model can store the structural information and parameter values of the trained image processing model locally. If the two are different, the execution subject for training the image processing model can send the structural information and parameter values of the trained image processing model to the execution subject for image processing.
According to the analysis, the preset eye object is not displayed in the first sample image, the preset eye object is displayed in the second sample image, and in the process of training the image processing model, the first sample image included in the sample is taken as input of the initial model, and the second sample image included in the sample is taken as expected output of the initial model. Therefore, when the preset eye object is not displayed in the eye image to be processed, the trained image processing model can be used for adding the preset eye object in the region corresponding to the preset eye object in the eye image to be processed. Thereby realizing a model trained by a machine learning method, adding a preset eye object, for example, at least one of double eyelid and lying silkworm, to an eye image to be processed.
In some optional implementations of this embodiment, on the basis of the sample set, a second image of a sample included in the sample set may be used as an input of the initial model, and a first image of a sample corresponding to the input second image of the sample may be used as an expected output of the initial model, so as to train to obtain the image processing model.
Here, the difference from the alternative implementations described above is that in training the image processing model, the input and desired output of the initial model are reversed. Thus, the trained image processing model can be used for removing the preset eye objects in the eye image to be processed. And further, the model obtained through training by a machine learning method is realized, and preset eye objects displayed in the eye image to be processed are removed, for example, at least one of displayed double eyelid and lying silkworm is removed.
In some alternative implementations of the present embodiment, the samples for training the image processing model described above may be obtained by the following steps.
The first step is to cut out an eye image from a first face image in the acquired first face image set to obtain a first eye image set.
The executing body of the training image processing model may obtain the first set of face images from a local or communication connection database. Wherein the first face image is typically an image with a face displayed.
After the first face image set is acquired, the execution body of the training image processing model may cut out an eye image from each first face image or a part of the first face images, so as to obtain the first eye image set. It can be understood that the first eye image is an eye image cut out from the first face image.
And secondly, horizontally overturning the first eye image in the first eye image set to obtain a second eye image set.
After the first eye image set is obtained, the execution main body of the training image processing model can horizontally overturn each or part of the first eye images, so that a second eye image set is obtained. It can be understood that the second eye image is the image obtained after the first eye image is flipped.
Here, horizontally inverting the first eye image generally means that the first eye image is symmetrical with respect to the vertical axis. For example, after the first eye image displaying the left eye of the person is flipped over, the second eye image displaying the right eye of the person may be obtained. It will be appreciated that by flipping the first eye image, the number of samples of the training image processing model may be increased.
And thirdly, selecting an eye image, in which the preset eye object does not exist, from the first eye image set and the second eye image set, in the area corresponding to the preset eye object, and obtaining a third eye image set.
In general, an execution subject of the training image processing model may divide an eye image in which a preset eye object is displayed and an eye image in which a preset eye object is not displayed from the first eye image set and the second eye image set by an image classification model trained in advance. Therefore, the execution main body of the training image processing model can randomly select the eye image of the preset eye object not to be displayed, and a third eye image set is obtained. It can be understood that the third eye image is the selected eye image that does not display the preset eye object.
It should be noted that the image classification model may be an image classification model constructed by a convolutional neural network, which is not described herein.
And a fourth step of inputting the third eye images in the third eye image set into a pre-trained sample processing model to obtain a fourth eye image set. The sample processing model is used for adding a preset eye object to the third eye image. It can be understood that the fourth eye image is an image obtained after adding the preset eye object to the third eye image. That is, in the fourth eye image and the third eye image, the difference in the other areas except for the area corresponding to the preset eye object is small.
The initial model of the training sample processing model may be an countermeasure generation network for processing the eye image, for example, a cyclic countermeasure generation network (Cycle Generative Adversarial Nets). Unlike the initial model, in which the image processing model is trained, there is typically a large difference between the image as input and the image as desired output in the same sample during training, and the eye sites displayed by the two are typically from different faces. The training process of the sample processing model is similar to that of the image processing model, and will not be repeated here.
And a fifth step of generating samples in the sample set based on the fourth eye image selected from the fourth eye image set and the corresponding third eye image selected from the third eye image set.
After the fourth eye image set is obtained, the execution subject of the training image processing model may randomly select the fourth eye image therefrom or select the fourth eye image according to the operation of the user. In addition, the execution subject of the training image processing model may select a third eye image corresponding to the selected fourth eye image from the third eye image set. Here, the third eye image corresponding to the selected fourth eye image is the third eye image that can be obtained after being processed by the sample processing model.
After selecting the fourth eye image and the corresponding third eye image, the execution subject of the training image processing model may combine the fourth eye image and the corresponding third eye image into one sample. Thus, a sample set can be obtained by multiple selections. Thus, methods of generating samples for training the above-described image processing model are enriched. In practice, an eye image with a preset eye object in a region corresponding to the preset eye object may be selected from the first eye image set and the second eye image set, and then the preset eye object displayed in the selected eye image is removed through a sample processing model, so that a sample for training the image processing model is obtained by adopting a method similar to the method in the fifth step.
Step 203, replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing.
In this embodiment, after the post-processing eye image is obtained, the execution body may replace the to-be-processed eye image in the to-be-processed face image with the post-processing eye image, so as to generate the post-processing face image. In general, a face displayed in a face image to be processed has a certain offset angle, and accordingly, there is an angular deviation between an eye image to be processed in the face image to be processed and the face image to be processed. Therefore, the execution body can rotate the processed eye image by a certain angle through affine transformation, and then replace the eye image to be processed in the face image to be processed with the processed eye image.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the image processing method according to the present embodiment. In the application scenario of fig. 3, first, the server 301 may acquire a face image 302 to be processed from a terminal device (not shown in the figure) connected in communication. Then, the server 301 may determine the eye image 303 to be processed and the eye image 304 to be processed from the face image 302 to be processed. After that, the server 301 may input the determined eye image to be processed 303 and the eye image to be processed 304 into the image processing model 305, respectively, to obtain the eye image after processing 306 and the eye image after processing 307, respectively. Then, the server 301 may replace the eye images to be processed in the face image to be processed 302 with the obtained post-processing eye images 306 and 307, respectively, thereby generating post-processing face images.
Currently, in adding double eyelid or lying silkworm to a face image, one of the prior arts, as described in the background of the present disclosure, is implemented by a preset map. Therefore, the method has no pertinence to different face images. In general, the difference between eye positions displayed in different face images is large. In addition, placing a pre-set map in a face image tends to cause an uncoordinated ratio with the eye positions in the face image. By means of the method provided by the embodiment of the invention, different face images to be processed can be correspondingly processed through the pre-trained image processing model, and targeted addition of double eyelid or lying silkworm in the face images to be processed is achieved, so that incoordination between the added double eyelid or lying silkworm in the face images to be processed and eye parts is avoided. In addition, the method provided by the embodiment of the present disclosure may further implement targeted removal of double eyelid or lying silkworm displayed in the face image to be processed.
With further reference to fig. 4, a flow 400 of yet another embodiment of an image processing method is shown. The flow 400 of the image processing method comprises the steps of:
Step 401, determining an eye image to be processed from the acquired face image to be processed.
In this embodiment, the execution body of the image processing method (such as the server 104 shown in fig. 1) may acquire a face image sent by a terminal device connected in communication as a face image to be processed.
Step 402, inputting the eye image to be processed into a pre-trained image processing model to obtain a processed eye image.
Step 403, replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing.
The steps 401, 402, and 403 may be performed in a similar manner as the steps 201, 202, and 203 in the embodiment shown in fig. 2, and the descriptions of the steps 201, 202, and 203 are also applicable to the steps 401, 402, and 403, which are not repeated herein.
And step 404, the processed face image is sent to the terminal equipment in communication connection, so that the terminal equipment displays the processed face image.
In this embodiment, the execution body may send the generated processed face image to a terminal device (such as terminal devices 101, 102 shown in fig. 1) connected in communication. Generally, after receiving the processed face image sent by the execution subject, the terminal device may display the processed face image.
As can be seen from fig. 4, compared with the embodiment corresponding to fig. 2, the flow 400 of the image processing method in this embodiment represents the step of sending the processed face image to the communicatively connected terminal device. Thus, in the solution described in this embodiment, after the terminal device sends the face image captured by the local or the camera installed thereon to the execution subject, the execution subject may return the processed face image generated for the face image. Thereby, at least one of double eyelid and lying silkworm is added to the face image uploaded by the user, or at least one of double eyelid and lying silkworm in the face image uploaded by the user is removed.
With further reference to fig. 5, as an implementation of the method shown in the foregoing figures, the present disclosure provides an embodiment of an image processing apparatus, which corresponds to the method embodiment shown in fig. 2, and which is particularly applicable to various electronic devices.
As shown in fig. 5, the image processing apparatus 500 provided in the present embodiment includes a determination unit 501, a processing unit 502, and a generation unit 503. Wherein the determining unit 501 may be configured to: and determining an eye image to be processed from the acquired face image to be processed. The processing unit 502 may be configured to: and inputting the eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed. The generating unit 503 may be configured to: and replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing.
In the present embodiment, in the image processing apparatus 500: the specific processes of the determining unit 501, the processing unit 502 and the generating unit 503 and the technical effects thereof may refer to the descriptions related to step 201, step 202 and step 203 in the corresponding embodiment of fig. 2, and are not repeated herein.
In some optional implementations of this embodiment, the predetermined ocular object may include at least one of: double eyelid, lying silkworm.
In some optional implementations of the present embodiment, the image processing model may be obtained by training a sample set, where a sample in the sample set may be an image pair including a sample first image and a sample second image, a preset eye object does not exist in an area corresponding to the preset eye object in the sample first image, and a preset eye object exists in an area corresponding to the preset eye object in the sample second image.
In some optional implementations of this embodiment, the image processing model may be obtained through training as follows: taking a first sample image included in a sample in the sample set as an input of an initial model, taking a second sample image corresponding to the input first sample image as an expected output of the initial model, and training to obtain an image processing model.
In some optional implementations of this embodiment, the image processing model may be obtained through training as follows: taking a sample second image included in a sample in the sample set as an input of an initial model, taking a sample first image corresponding to the input sample second image as an expected output of the initial model, and training to obtain an image processing model.
In some optional implementations of this embodiment, the samples in the sample set may be obtained by: cutting out an eye image from a first face image in the acquired first face image set to obtain a first eye image set; performing horizontal overturning on the first eye images in the first eye image set to obtain a second eye image set; selecting an eye image, in which a preset eye object does not exist, from the first eye image set and the second eye image set in a region corresponding to the preset eye object, so as to obtain a third eye image set; inputting a third eye image in the third eye image set into a pre-trained sample processing model to obtain a fourth eye image set, wherein the sample processing model is used for adding a preset eye object to the third eye image; samples in the sample set are generated based on a fourth eye image selected from the fourth eye image set and a corresponding third eye image selected from the third eye image set.
In some optional implementations of this embodiment, the determining unit 501 may be further configured to: based on the key points extracted from the acquired face image to be processed, the eye image to be processed with the preset size is cut from the face image to be processed.
In some optional implementations of this embodiment, the apparatus 500 may further include: a transmitting unit (not shown in the figure). Wherein the transmitting unit may be configured to: and sending the processed face image to the terminal equipment in communication connection so that the terminal equipment displays the processed face image.
The device provided in the foregoing embodiment of the present disclosure may first determine, by the determining unit 501, an eye image to be processed from the acquired face image to be processed; then, the determined eye image to be processed can be input into a pre-trained image processing model through the processing unit 502, so as to obtain a processed eye image; then, the to-be-processed eye image in the to-be-processed face image may be replaced with the obtained post-processing eye image by the generating unit 503, to generate a post-processing face image. Therefore, the method and the device realize the targeted addition or removal of the preset eye objects to the face image to be processed.
Referring now to fig. 6, a schematic diagram of an electronic device (e.g., server in fig. 1) 600 suitable for use in implementing embodiments of the present disclosure is shown. The server illustrated in fig. 6 is merely an example, and should not be construed as limiting the functionality and scope of use of the embodiments of the present disclosure in any way.
As shown in fig. 6, the electronic device 600 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
In general, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, magnetic tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 shows an electronic device 600 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 6 may represent one device or a plurality of devices as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609, or from storage means 608, or from ROM 602. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing means 601. It should be noted that, the computer readable medium according to the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In an embodiment of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. Whereas in embodiments of the present disclosure, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the server; or may exist alone without being assembled into the server. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the server to: determining an eye image to be processed from the acquired face image to be processed; inputting an eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed; and replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing.
Computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments described in the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes a determination unit, a processing unit, and a generation unit. The names of these units do not limit the unit itself in some cases, and the determining unit may also be described as "a unit that determines an eye image to be processed from the acquired face image to be processed", for example.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention referred to in this disclosure is not limited to the specific combination of features described above, but encompasses other embodiments in which features described above or their equivalents may be combined in any way without departing from the spirit of the invention. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Claims (16)
1. An image processing method, comprising:
determining an eye image to be processed from the acquired face image to be processed;
inputting the eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed; the region corresponding to the preset eye object is a region for adding or removing the preset eye object in the eye image to be processed; the processed eye image is an image obtained after adding the preset eye object in the eye image to be processed, or an image obtained after removing the preset eye object displayed in the eye image to be processed; the preset ocular object comprises at least one of the following: double eyelid, lying silkworm;
And replacing the eye image to be processed in the face image to be processed with the eye image after processing to generate the face image after processing.
2. The method of claim 1, wherein the image processing model is trained using a set of samples, the samples in the set of samples being image pairs comprising a first image of the sample in which the pre-set ocular object is absent from a region corresponding to the pre-set ocular object and a second image of the sample in which the pre-set ocular object is present.
3. The method of claim 2, wherein the image processing model is trained by:
and taking a first sample image included in the samples in the sample set as an input of an initial model, taking a second sample image corresponding to the input first sample image as an expected output of the initial model, and training to obtain the image processing model.
4. The method of claim 2, wherein the image processing model is trained by:
and taking a sample second image included in the sample set as an input of an initial model, taking a sample first image corresponding to the input sample second image as an expected output of the initial model, and training to obtain the image processing model.
5. The method of claim 2, wherein the samples in the sample set are obtained by:
cutting out an eye image from a first face image in the acquired first face image set to obtain a first eye image set;
performing horizontal overturning on a first eye image in the first eye image set to obtain a second eye image set;
selecting an eye image in which the preset eye object does not exist in a region corresponding to the preset eye object from the first eye image set and the second eye image set, and obtaining a third eye image set;
inputting a third eye image in the third eye image set into a pre-trained sample processing model to obtain a fourth eye image set, wherein the sample processing model is used for adding the preset eye object to the third eye image;
samples in the sample set are generated based on a fourth eye image selected from the fourth eye image set and a corresponding third eye image selected from the third eye image set.
6. The method of claim 1, wherein the determining an eye image to be processed from the acquired face image to be processed comprises:
Based on key points extracted from the acquired face image to be processed, cutting out an eye image to be processed with a preset size from the face image to be processed.
7. The method of any of claims 1-6, wherein the method further comprises:
and sending the processed face image to a terminal device in communication connection so that the terminal device displays the processed face image.
8. An image processing apparatus comprising:
the determining unit is configured to determine an eye image to be processed from the acquired face image to be processed;
the processing unit is configured to input the eye image to be processed into a pre-trained image processing model to obtain a processed eye image, wherein the image processing model is used for processing an area corresponding to a preset eye object in the eye image to be processed; the region corresponding to the preset eye object is a region for adding or removing the preset eye object in the eye image to be processed; the processed eye image is an image obtained after adding the preset eye object in the eye image to be processed, or an image obtained after removing the preset eye object displayed in the eye image to be processed; the preset ocular object comprises at least one of the following: double eyelid, lying silkworm;
And the generating unit is configured to replace the eye image to be processed in the face image to be processed with the eye image after processing so as to generate the face image after processing.
9. The device of claim 8, wherein the image processing model is trained using a set of samples, the samples in the set of samples being image pairs comprising a first image of the sample in which the pre-set ocular object is absent from a region corresponding to the pre-set ocular object and a second image of the sample in which the pre-set ocular object is present.
10. The apparatus of claim 9, wherein the image processing model is trained by:
and taking a first sample image included in the samples in the sample set as an input of an initial model, taking a second sample image corresponding to the input first sample image as an expected output of the initial model, and training to obtain the image processing model.
11. The apparatus of claim 9, wherein the image processing model is trained by:
and taking a sample second image included in the sample set as an input of an initial model, taking a sample first image corresponding to the input sample second image as an expected output of the initial model, and training to obtain the image processing model.
12. The apparatus of claim 9, wherein the samples in the sample set are obtained by:
cutting out an eye image from a first face image in the acquired first face image set to obtain a first eye image set;
performing horizontal overturning on a first eye image in the first eye image set to obtain a second eye image set;
selecting an eye image in which the preset eye object does not exist in a region corresponding to the preset eye object from the first eye image set and the second eye image set, and obtaining a third eye image set;
inputting a third eye image in the third eye image set into a pre-trained sample processing model to obtain a fourth eye image set, wherein the sample processing model is used for adding the preset eye object to the third eye image;
samples in the sample set are generated based on a fourth eye image selected from the fourth eye image set and a corresponding third eye image selected from the third eye image set.
13. The apparatus of claim 8, wherein the determination unit is further configured to:
Based on key points extracted from the acquired face image to be processed, cutting out an eye image to be processed with a preset size from the face image to be processed.
14. The apparatus of any of claims 8-13, wherein the apparatus further comprises:
and the sending unit is configured to send the processed face image to a communication connected terminal device so as to enable the terminal device to display the processed face image.
15. A server, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.
16. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910409754.5A CN110136054B (en) | 2019-05-17 | 2019-05-17 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910409754.5A CN110136054B (en) | 2019-05-17 | 2019-05-17 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110136054A CN110136054A (en) | 2019-08-16 |
| CN110136054B true CN110136054B (en) | 2024-01-09 |
Family
ID=67574692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910409754.5A Active CN110136054B (en) | 2019-05-17 | 2019-05-17 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110136054B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110580678B (en) * | 2019-09-10 | 2023-06-20 | 北京百度网讯科技有限公司 | Image processing method and device |
| CN110766631A (en) * | 2019-10-21 | 2020-02-07 | 北京旷视科技有限公司 | Face image modification method and device, electronic equipment and computer readable medium |
| CN111462007B (en) * | 2020-03-31 | 2023-06-09 | 北京百度网讯科技有限公司 | Image processing method, device, equipment and computer storage medium |
| CN112381709B (en) * | 2020-11-13 | 2022-06-21 | 北京字节跳动网络技术有限公司 | Image processing method, model training method, device, equipment and medium |
| CN112465717A (en) * | 2020-11-25 | 2021-03-09 | 北京字跳网络技术有限公司 | Face image processing model training method and device, electronic equipment and medium |
| CN112489169B (en) * | 2020-12-17 | 2024-02-13 | 脸萌有限公司 | Portrait image processing method and device |
| CN113344776B (en) * | 2021-06-30 | 2023-06-27 | 北京字跳网络技术有限公司 | Image processing method, model training method, device, electronic equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107153805A (en) * | 2016-03-02 | 2017-09-12 | 北京美到家科技有限公司 | Customize makeups servicing unit and method |
| CN107993209A (en) * | 2017-11-30 | 2018-05-04 | 广东欧珀移动通信有限公司 | Image processing method, device, computer-readable recording medium and electronic equipment |
| CN108021905A (en) * | 2017-12-21 | 2018-05-11 | 广东欧珀移动通信有限公司 | image processing method, device, terminal device and storage medium |
| CN108986016A (en) * | 2018-06-28 | 2018-12-11 | 北京微播视界科技有限公司 | Image beautification method, device and electronic equipment |
| CN109241930A (en) * | 2018-09-20 | 2019-01-18 | 北京字节跳动网络技术有限公司 | Method and apparatus for handling supercilium image |
| CN109584153A (en) * | 2018-12-06 | 2019-04-05 | 北京旷视科技有限公司 | Modify the methods, devices and systems of eye |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108229508B (en) * | 2016-12-15 | 2022-01-04 | 富士通株式会社 | Training apparatus and training method for training image processing apparatus |
-
2019
- 2019-05-17 CN CN201910409754.5A patent/CN110136054B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107153805A (en) * | 2016-03-02 | 2017-09-12 | 北京美到家科技有限公司 | Customize makeups servicing unit and method |
| CN107993209A (en) * | 2017-11-30 | 2018-05-04 | 广东欧珀移动通信有限公司 | Image processing method, device, computer-readable recording medium and electronic equipment |
| CN108021905A (en) * | 2017-12-21 | 2018-05-11 | 广东欧珀移动通信有限公司 | image processing method, device, terminal device and storage medium |
| CN108986016A (en) * | 2018-06-28 | 2018-12-11 | 北京微播视界科技有限公司 | Image beautification method, device and electronic equipment |
| CN109241930A (en) * | 2018-09-20 | 2019-01-18 | 北京字节跳动网络技术有限公司 | Method and apparatus for handling supercilium image |
| CN109584153A (en) * | 2018-12-06 | 2019-04-05 | 北京旷视科技有限公司 | Modify the methods, devices and systems of eye |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110136054A (en) | 2019-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110136054B (en) | Image processing method and device | |
| CN107578017B (en) | Method and apparatus for generating image | |
| CN108830235B (en) | Method and apparatus for generating information | |
| US11436863B2 (en) | Method and apparatus for outputting data | |
| CN109981787B (en) | Method and device for displaying information | |
| CN110059623B (en) | Method and apparatus for generating information | |
| CN110288705B (en) | Method and device for generating three-dimensional model | |
| CN110070076B (en) | Method and device for selecting training samples | |
| CN108510084B (en) | Method and apparatus for generating information | |
| US11232560B2 (en) | Method and apparatus for processing fundus image | |
| CN112820408A (en) | Surgical operation risk determination method, related device and computer program product | |
| CN107622241A (en) | Display methods and device for mobile device | |
| CN116934577A (en) | Method, device, equipment and medium for generating style image | |
| CN108921138B (en) | Method and apparatus for generating information | |
| CN109949213B (en) | Method and apparatus for generating image | |
| CN112183388A (en) | Image processing method, apparatus, device and medium | |
| CN111258414B (en) | Method and device for adjusting screen | |
| EP3921747A1 (en) | Systems and methods for item response modelling of digital assessments | |
| CN112967299B (en) | Image cropping method and device, electronic equipment and computer readable medium | |
| CN111586295B (en) | Image generation method and device and electronic equipment | |
| CN115170714A (en) | Scanned image rendering method and device, electronic equipment and storage medium | |
| CN111314627B (en) | Method and apparatus for processing video frames | |
| CN111260756B (en) | Method and device for transmitting information | |
| CN110381374B (en) | Image processing method and device | |
| CN111914861A (en) | Target detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |