CN115456906A - Damaged image-based facial image restoration method, device, equipment and medium - Google Patents
Damaged image-based facial image restoration method, device, equipment and medium Download PDFInfo
- Publication number
- CN115456906A CN115456906A CN202211189288.2A CN202211189288A CN115456906A CN 115456906 A CN115456906 A CN 115456906A CN 202211189288 A CN202211189288 A CN 202211189288A CN 115456906 A CN115456906 A CN 115456906A
- Authority
- CN
- China
- Prior art keywords
- image
- face
- key point
- cutting
- repairing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000001815 facial effect Effects 0.000 title claims description 30
- 230000009466 transformation Effects 0.000 claims abstract description 60
- 230000008439 repair process Effects 0.000 claims abstract description 46
- 238000012937 correction Methods 0.000 claims abstract description 31
- 239000011159 matrix material Substances 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 19
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims abstract description 16
- 238000001514 detection method Methods 0.000 claims description 11
- 238000013528 artificial neural network Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 241001292396 Cirrhitidae Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/02—Affine transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The application relates to a method, a device, equipment and a medium for repairing a face image based on a damaged image, wherein the method comprises the following steps: acquiring an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image; detecting key points of the face of the preliminary restored image to obtain a key point image of the face; clipping and rotating alignment processing are carried out on the face key point image to obtain a face clipping image; obtaining a transformation matrix based on the key point position difference between the face clipping image and the face key point image, and carrying out affine transformation on the clipping face image according to the transformation matrix to obtain a face correction image; and carrying out depth restoration on the face correction image to obtain a depth restoration result, and generating a target restoration image based on the depth restoration result. The invention realizes the accurate restoration of the face image and improves the face restoration efficiency.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, a device, and a medium for repairing a face image based on a damaged image.
Background
In the process of image acquisition, transmission and storage, there are many factors that can cause image information to be lost or damaged, and in order to repair damaged photos, many semi-automatic repair software is available in the market. However, these software often cannot automatically identify the damaged area for repairing, and only the final repairing result can be obtained through a manual interaction mode, and the repairing work related to the face area is more difficult. The manual repair of a damaged photo usually requires a lot of manpower, so it is a considerable matter to research a more efficient and convenient method for repairing a damaged photo.
The existing face repairing method is divided into a traditional method and a deep learning method. Traditional methods typically solve the inverse problem based on a degenerate model and manual prior, which shows limited performance in practical applications. In recent years, deep Neural Networks (DNNs) have shown superior results in various computer vision tasks, and many DNN-based face restoration methods have been developed and show better performance than conventional methods. Direct training of Deep Neural Networks (DNNs) often does not yield satisfactory results due to the high prevalence of the problem and the complex location degradation. In order to recover High Quality (HQ) face images with realistic textures from low quality face images, many spatial transform-based networks have been proposed that show impressive results on artificially synthesized low quality faces; however, they cannot handle real world low quality facial images. Existing methods based on generation of countermeasure networks (GANs) can achieve better and more realistic results, but tend to produce overly smooth results. Therefore, when the existing face image restoration method faces damaged and folded images, the face image cannot be accurately restored, and the face restoration efficiency cannot be improved.
Disclosure of Invention
The embodiment of the application aims to provide a method, a device, equipment and a medium for repairing a face image based on a damaged image, so that the face image can be accurately repaired, and the efficiency of face repair is improved.
In order to solve the above technical problem, an embodiment of the present application provides a face image repairing method based on a damaged image, including:
acquiring an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image;
face key point detection is carried out on the preliminary repairing image to obtain a face key point image;
cutting and rotationally aligning the face key point image to obtain a face cutting image, wherein the face cutting image comprises face cutting key points;
obtaining a transformation matrix based on the key point position difference between the face cutting image and the face key point image, and carrying out affine transformation on the cutting face image according to the transformation matrix to obtain a face correction image;
and performing depth restoration on the face correction image to obtain a depth restoration result, and generating a target restoration image based on the depth restoration result.
In order to solve the above technical problem, an embodiment of the present application provides a facial image restoration device based on a damaged image, including:
the device comprises a to-be-processed image acquisition module, a scratch detection module and a repair module, wherein the to-be-processed image acquisition module is used for acquiring an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image;
the face key point detection module is used for detecting face key points of the preliminary restoration image to obtain a face key point image;
the face cutting image generation module is used for cutting and rotationally aligning the face key point image to obtain a face cutting image, wherein the face cutting image comprises face cutting key points;
the correction image generation module is used for obtaining a transformation matrix based on the key point position difference between the human face cutting image and the human face key point image, and carrying out affine transformation on the cutting human face image according to the transformation matrix to obtain a human face correction image;
and the target restoration image generation module is used for performing depth restoration on the face correction image to obtain a depth restoration result and generating a target restoration image based on the depth restoration result.
In order to solve the technical problems, the invention adopts a technical scheme that: a computer device is provided that includes, one or more processors; the storage is used for storing one or more programs, so that the one or more processors can realize the facial image restoration method based on the damaged image.
In order to solve the technical problems, the invention adopts a technical scheme that: a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method for repairing a facial image based on a damaged image as described in any one of the above.
The embodiment of the invention provides a method, a device, equipment and a medium for repairing a face image based on a damaged image. The method comprises the following steps: acquiring an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image; detecting key points of the face of the preliminary restored image to obtain a key point image of the face; cutting and rotationally aligning the face key point image to obtain a face cutting image, wherein the face cutting image comprises face cutting key points; obtaining a transformation matrix based on the key point position difference between the face clipping image and the face key point image, and performing affine transformation on the clipping face image according to the transformation matrix to obtain a face correction image; and carrying out depth restoration on the face correction image to obtain a depth restoration result, and generating a target restoration image based on the depth restoration result. The embodiment of the invention firstly detects and repairs the damaged area of the image to be processed, realizes the initial repair of the damaged image, then cuts the face area in the image to obtain the cut face image, and further repairs the cut face image, thereby obtaining the target repaired image, being beneficial to improving the precision of face repair, and respectively repairs the damaged area and the face, avoiding generating the over-smooth effect, and being beneficial to improving the repair efficiency.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of an implementation of a flow of a damaged image-based face image repairing method according to an embodiment of the present application;
fig. 2 is a flowchart of another implementation of a sub-process in a method for repairing a face image based on a damaged image according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a comparison between a damaged image and a scratched area provided by an embodiment of the present application;
FIG. 4 is a schematic diagram of a comparison between a damaged image and a preliminary repaired image provided by an embodiment of the present application;
fig. 5 is a flowchart of another implementation of a sub-process in a method for repairing a face image based on a damaged image according to an embodiment of the present application;
fig. 6 is a flowchart of another implementation of a sub-process in a method for repairing a face image based on a damaged image according to an embodiment of the present application;
fig. 7 is a schematic diagram illustrating comparison between before and after face region restoration provided in an embodiment of the present application;
FIG. 8 is a schematic diagram illustrating a comparison of a damaged image and a target repair image provided by an embodiment of the present application;
FIG. 9 is a schematic diagram of a facial image restoration apparatus based on a damaged image according to an embodiment of the present application;
fig. 10 is a schematic diagram of a computer device provided in an embodiment of the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
The present invention will be described in detail below with reference to the drawings and embodiments.
The facial image restoration method based on the damaged image provided by the embodiment of the present application is generally executed by a server, and accordingly, the facial image restoration apparatus based on the damaged image is generally configured in the server.
Referring to fig. 1, fig. 1 shows an embodiment of a facial image restoration method based on a damaged image.
It should be noted that, if the result is substantially the same, the method of the present invention is not limited to the flow sequence shown in fig. 1, and the method includes the following steps:
s1: the method comprises the steps of obtaining an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image.
Specifically, the image to be processed refers to an image that needs to be subjected to face restoration, and may be a folded image or a broken image. The scratch area of the image to be processed is detected to obtain the scratch area, the scratch area is repaired to obtain a primary repair image, the image to be processed is subjected to primary repair, follow-up repair of the face area is facilitated, and the accuracy and efficiency of repair are improved.
Referring to fig. 2 to 4, fig. 2 shows an embodiment of step S1, and fig. 3 is a schematic diagram illustrating a comparison between a damaged image and a scratched area provided in an embodiment of the present disclosure; fig. 4 is a schematic diagram of a comparison between a damaged image and a preliminary repair image provided in the embodiment of the present application. Details of step S1 are as follows:
s11: and acquiring an image to be processed, and detecting a scratch area of the image to be processed to obtain a scratch area mask.
S12: and performing expansion treatment on the mask of the scratch area to obtain the scratch area.
S13: and repairing the scratched area by adopting a preset repairing mode to obtain a primary repairing image.
Specifically, directly restoreing to a damaged image, often hardly obtaining ideal effect, consequently this application embodiment proposes earlier to detect damaged part in the image, also promptly detects the scratch area of handling the image, obtains scratch area mask, then carries out inflation processing to scratch area mask, makes the mask can better cover original scratch area to obtain scratch area. The method of scratch detection may use the method of Halcon surface scratch detection. And finally, repairing the scratched area by adopting a preset repairing mode to obtain a primary repairing image. The preset restoration mode can adopt an open-source image watermarking removing algorithm. The algorithm for removing the watermark of the open-source image can remove the watermark and also can remove scratches and damages, so that the effect similar to restoration is obtained.
S2: and detecting the key points of the face of the primary restored image to obtain a key point image of the face.
Specifically, the above steps have already performed the preliminary repair of the scratch region on the image to be processed, and the subsequent steps mainly aim at the deep repair of the face region to improve the face repair effect. In the embodiment of the application, in order to obtain a good face repairing effect, the position of the face in the image needs to be acquired at first, so that the interference of other information can be eliminated, and the face information can be better recovered. Therefore, in the embodiment, the facexlib library is called to perform face key point detection on the face image, so as to obtain the face key point image. The face key point image at least comprises left eyes, right eyes, a nose, left and right mouth corners, and the leftmost and rightmost key points of the face.
S3: and (4) clipping and rotating and aligning the face key point image to obtain a face clipping image, wherein the face clipping image comprises face clipping key points.
Specifically, the face cropping key points include a left eye key point, a right eye key point, a left mouth key point, a right mouth key point, a leftmost key point, and a rightmost key point.
Referring to fig. 5, fig. 5 shows an embodiment of step S3, which is described in detail as follows:
s31: and determining the cutting width in the vertical direction based on the left eye key point, the right eye key point, the left mouth key point and the right mouth key point.
Referring to fig. 6, fig. 6 shows an embodiment of step S31, which is described in detail as follows:
s311: and acquiring the central positions of the key points of the left eye and the right eye as the central positions of the two eyes.
S312: and acquiring the central positions of the key points of the left mouth and the right mouth as the central positions of the mouths.
S313: and calculating the distance between the center positions of the two eyes and the center position of the mouth to obtain the middle cutting width in the vertical direction, and respectively calculating the middle cutting width, the bottom cutting width and the top cutting width in the vertical direction on the basis of the middle cutting width, the center positions of the two eyes and the center position of the mouth.
Specifically, the center positions of the key points of the left eye and the right eye are obtained as the center positions of both eyes, and the center positions of the key points of the left mouth and the right mouth are obtained as the center positions of the mouths, then the distance between the center positions of both eyes and the center position of the mouth is calculated, and the distance is used as the middle cutting width in the vertical direction. Then, a preset ratio of the middle cutting width to the cutting width in the vertical direction is obtained, for example, the middle cutting width is 35, the preset ratio is 35%, and if the preset ratios of the bottom cutting width and the top cutting width are 35% and 30%, respectively, the bottom cutting width and the top cutting width are 35 and 30, respectively.
Specifically, the distance calculated in the embodiment of the present application refers to a pixel distance.
Further, an embodiment of step S31 is provided, which is detailed as follows:
calculating the distance between the center positions of the two eyes and the center position of the mouth to obtain the cutting width of the middle part;
according to the middle cutting width, calculating a first preset proportional distance from the center position of the mouth to the vertical direction, and obtaining a bottom cutting width;
and calculating a second preset proportional distance from the center positions of the two eyes to the vertical direction upwards according to the middle cutting width to obtain the top cutting width.
In particular, the embodiment of the present application requires defining the dividing lines of the middle cutting width, the bottom cutting width and the top cutting width. The middle cutting width can define a dividing line by the central point of key points of two eyes and the central point of a mouth; the middle cutting width is used for defining a dividing line by a first preset proportional distance from the center of the mouth to the vertical direction; the top cutting width is to define a dividing line by a second preset proportional distance from the center position of the two eyes to the vertical direction.
It should be noted that the first preset ratio and the second preset ratio are set according to actual situations, and are not limited herein. In one embodiment, the first predetermined ratio and the second predetermined ratio are 35% and 30%, respectively.
S314: and obtaining the cutting width in the vertical direction based on the middle cutting width, the bottom cutting width and the top cutting width.
Specifically, the middle cutting width, the bottom cutting width, and the top cutting width are obtained in the above steps, and then the middle cutting width, the bottom cutting width, and the top cutting width are combined together to be used as the cutting width in the vertical direction.
S32: and determining a cutting central point in the horizontal direction according to the leftmost key point and the rightmost key point.
Specifically, in the horizontal direction of the face area, the center positions of the leftmost key point and the rightmost key point are taken as the clipping center points in the horizontal direction. The leftmost key point and the rightmost key point refer to two key points which are positioned on the leftmost side and the rightmost side in the face area.
S33: and according to the cutting width in the vertical direction and the cutting central point in the horizontal direction, cutting the key point image of the face in a mode that the cutting width is consistent with the cutting height to obtain an initial cutting image.
Specifically, after the cutting width in the vertical direction and the cutting center point in the horizontal direction are determined, the face key point image is cut according to the mode that the cutting width and the cutting height are consistent, and an initial cutting image is obtained. The initial cut image is a face image with the same length and width.
S34: and carrying out rotation alignment processing on the initial cutting image based on the face key point image to obtain a face cutting image.
Specifically, since the direction of the initial cropped image after cropping may be shifted in the horizontal or vertical direction, the initial cropped image is subjected to the rotation alignment processing with the face key point image as a reference, so as to obtain the face cropped image.
S4: and performing affine transformation on the cut face image according to the transformation matrix to obtain a face correction image.
Specifically, the above steps have been performed on the face clipping image by rotation alignment, but in the clipping process, the positions of the key points may also be shifted, so in the embodiment of the present application, a transformation matrix is obtained based on the difference between the positions of the key points of the face clipping image and the face key point image, and affine transformation is performed on the clipping face image according to the transformation matrix, so as to obtain a face correction image.
Further, an embodiment of step S4 is provided, which is detailed as follows:
and respectively acquiring the positions of key points in the face cutting image and the face key point image to obtain the positions of the cutting key points and the positions of the marking key points.
And performing corresponding subtraction processing on the positions of the cutting key points and the positions of the labeling key points to obtain a key point position difference, and constructing a transformation matrix based on the key point position difference.
And performing affine transformation on the cut face image according to the transformation matrix to obtain a face correction image.
Specifically, key point positions, cutting key point positions and marking key point positions in the face cutting image and the face key point image are respectively obtained. And only obtaining key points of the face key point image corresponding to the face within the range of the face cutting image. And then, carrying out one-to-one correspondence on the positions of the cutting key points and the positions of the marking key points, if the positions of the right-eye key points in the cutting key points correspond to the positions of the right-eye key points in the marking key points, then carrying out subtraction processing on the corresponding key points to obtain key point position differences, then constructing a transformation matrix based on the key point position differences, and finally carrying out affine transformation on the cut face image according to the transformation matrix to obtain a face correction image.
S5: and carrying out depth restoration on the face correction image to obtain a depth restoration result, and generating a target restoration image based on the depth restoration result.
Referring to fig. 7 and 8, fig. 7 is a schematic diagram illustrating comparison between before and after face region restoration according to an embodiment of the present application; fig. 8 is a schematic diagram of a comparison between a damaged image and a target repair image provided in the embodiment of the present application.
Further, an embodiment of step S5 is provided, which is detailed as follows:
performing deep restoration on the face correction image based on the deep neural network and the generated confrontation network to obtain a deep restoration result;
carrying out inverse transformation processing on the depth restoration result to obtain an inverse transformation image;
and identifying the face area in the primary repairing image, and pasting the inverse transformation image back to the face area to obtain a target repairing image.
In the embodiment of the application, the face correction image is subjected to deep restoration based on the deep neural network and the generated confrontation network, and a deep restoration result is obtained. For example, a generated confrontation network GAN for generating a high-quality face image is trained, the generated confrontation network GAN is embedded into a deep neural network DNN, the generated confrontation network GAN is embedded into the deep neural network DNN by using a group of synthesized low-quality face images for fine adjustment to obtain a deep repair model, and the deep repair model is used for deep repair of a face correction image to obtain a deep repair result. And then carrying out inverse transformation processing on the depth repairing result to obtain an inverse transformation image. Wherein, the inverse transformation image is a face image subjected to depth restoration. And finally, recognizing the face area in the primary repairing image, and pasting the inverse transformation image back to the face area to obtain a target repairing image.
In the embodiment, an image to be processed is obtained, a scratch area of the image to be processed is detected to obtain the scratch area, and the scratch area is repaired to obtain a primary repair image; detecting key points of the face of the preliminary restored image to obtain a key point image of the face; cutting and rotationally aligning the face key point image to obtain a face cutting image, wherein the face cutting image comprises face cutting key points; obtaining a transformation matrix based on the key point position difference between the face clipping image and the face key point image, and carrying out affine transformation on the clipping face image according to the transformation matrix to obtain a face correction image; and carrying out depth restoration on the face correction image to obtain a depth restoration result, and generating a target restoration image based on the depth restoration result. The embodiment of the invention firstly detects and repairs the damaged area of the image to be processed, realizes the initial repair of the damaged image, then cuts the face area in the image to obtain the cut face image, and further repairs the cut face image, thereby obtaining the target repaired image, being beneficial to improving the precision of face repair, and respectively repairs the damaged area and the face, avoiding generating the over-smooth effect, and being beneficial to improving the repair efficiency.
Referring to fig. 9, as an implementation of the method shown in fig. 1, the present application provides an embodiment of a facial image restoration apparatus based on a damaged image, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 1, and the apparatus may be applied to various electronic devices.
As shown in fig. 9, the facial image restoration apparatus based on a damaged image of the present embodiment includes: a pending image acquisition module 61, a face key point detection module 62, a face clipping image generation module 63, a correction image generation module 64 and a target restoration image generation module 65, wherein:
the to-be-processed image acquisition module 61 is used for acquiring the to-be-processed image, detecting a scratch area of the to-be-processed image to obtain the scratch area, and repairing the scratch area to obtain a primary repair image;
a face key point detection module 62, configured to perform face key point detection on the preliminary restored image to obtain a face key point image;
the face cropping image generation module 63 is configured to crop and rotationally align the face key point image to obtain a face cropping image, where the face cropping image includes face cropping key points;
a corrected image generation module 64, configured to obtain a transformation matrix based on a difference between the key point positions of the face cut image and the face key point image, and perform affine transformation on the cut face image according to the transformation matrix to obtain a face corrected image;
and the target restoration image generation module 65 is configured to perform depth restoration on the face correction image to obtain a depth restoration result, and generate a target restoration image based on the depth restoration result.
Further, the face cropping image generation module 63 includes:
the cutting width determining submodule is used for determining the cutting width in the vertical direction based on the left-eye key point, the right-eye key point, the left-mouth key point and the right-mouth key point;
the cutting center point determining submodule is used for determining a cutting center point in the horizontal direction according to the leftmost key point and the rightmost key point;
the initial cutting image generation submodule is used for cutting the key point image of the face according to the cutting width in the vertical direction and the cutting central point in the horizontal direction in a mode of consistent cutting width and height to obtain an initial cutting image;
and the rotation alignment processing submodule is used for performing rotation alignment processing on the initial cutting image based on the face key point image to obtain a face cutting image.
Further, the cropping width determination submodule includes:
the two-eye center position acquisition unit is used for acquiring the center positions of the key points of the left eye and the right eye as the center positions of the two eyes;
a mouth center position acquisition unit configured to acquire a center position of the left mouth key point and the right mouth key point as a mouth center position;
the width calculation unit is used for calculating the distance between the center positions of the two eyes and the center position of the mouth to obtain the middle cutting width in the vertical direction, and respectively calculating the middle cutting width, the bottom cutting width and the top cutting width in the vertical direction on the basis of the middle cutting width, the center positions of the two eyes and the center position of the mouth;
and the cutting width determining unit is used for obtaining the cutting width in the vertical direction based on the middle cutting width, the bottom cutting width and the top cutting width.
Further, the width calculation unit includes:
the middle cutting width generating subunit is used for calculating the distance between the center positions of the two eyes and the center position of the mouth to obtain the middle cutting width;
the bottom cutting width generating subunit is used for calculating a first preset proportional distance from the center position of the mouth to the vertical direction according to the middle cutting width to obtain a bottom cutting width;
and the top cutting width generating subunit is used for calculating a second preset proportional distance from the center positions of the two eyes to the vertical direction according to the middle cutting width to obtain the top cutting width.
Further, the rectified image generation module 64 includes:
the key point position acquisition submodule is used for respectively acquiring key point positions in the face cutting image and the face key point image to obtain a cutting key point position and a marking key point position;
the transformation matrix construction submodule is used for carrying out corresponding subtraction processing on the positions of the cutting key points and the positions of the marking key points to obtain key point position differences and constructing a transformation matrix based on the key point position differences;
and the affine transformation submodule is used for carrying out affine transformation on the cut face image according to the transformation matrix to obtain a face correction image.
Further, the target repair image generation module 65 includes:
the depth restoration result generation submodule is used for carrying out depth restoration on the face correction image based on the depth neural network and the generated confrontation network to obtain a depth restoration result;
the inverse transformation image generation submodule is used for carrying out inverse transformation processing on the depth restoration result to obtain an inverse transformation image;
and the image pasting sub-module is used for identifying the face area in the primary repairing image and pasting the inverse transformation image back to the face area to obtain the target repairing image.
Further, the to-be-processed image acquiring module 61 includes:
the scratch area mask acquisition submodule is used for acquiring an image to be processed and detecting a scratch area of the image to be processed to obtain a scratch area mask;
the scratch area generation submodule is used for performing expansion processing on the mask of the scratch area to obtain the scratch area;
and the scratch area repairing submodule is used for repairing the scratch area by adopting a preset repairing mode to obtain a primary repairing image.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 10, fig. 10 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 7 comprises a memory 71, a processor 72, a network interface 73, communicatively connected to each other by a system bus. It is noted that only a computer device 7 having three components memory 71, processor 72, network interface 73 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance, and the hardware thereof includes but is not limited to a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 71 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 71 may be an internal storage unit of the computer device 7, such as a hard disk or a memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device 7. Of course, the memory 71 may also comprise both an internal storage unit of the computer device 7 and an external storage device thereof. In the present embodiment, the memory 71 is generally used for storing an operating system installed in the computer device 7 and various types of application software, such as program codes of a damaged image-based face image restoration method. Further, the memory 71 may also be used to temporarily store various types of data that have been output or are to be output.
The network interface 73 may comprise a wireless network interface or a wired network interface, and the network interface 73 is typically used to establish a communication connection between the computer device 7 and other electronic devices.
The present application further provides another embodiment, which is to provide a computer-readable storage medium, where a computer program is stored, and the computer program can be executed by at least one processor, so that the at least one processor executes the steps of the method for repairing a face image based on a damaged image.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method of the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and the embodiments are provided so that this disclosure will be thorough and complete. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields, and all the equivalent structures are within the protection scope of the present application.
Claims (10)
1. A facial image restoration method based on damaged images is characterized by comprising the following steps:
acquiring an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image;
detecting key points of the face of the preliminary repairing image to obtain a key point image of the face;
cutting and rotationally aligning the face key point image to obtain a face cutting image, wherein the face cutting image comprises face cutting key points;
obtaining a transformation matrix based on the key point position difference between the face cutting image and the face key point image, and carrying out affine transformation on the cutting face image according to the transformation matrix to obtain a face correction image;
and performing depth restoration on the face correction image to obtain a depth restoration result, and generating a target restoration image based on the depth restoration result.
2. The method for repairing facial image based on damaged image according to claim 1, wherein the facial clipping key points include a left eye key point, a right eye key point, a left mouth key point, a right mouth key point, a leftmost key point and a rightmost key point, and the clipping and rotation aligning the facial key point image to obtain the facial clipping image comprises:
determining a cutting width in a vertical direction based on the left-eye key point, the right-eye key point, the left-mouth key point and the right-mouth key point;
determining a cutting central point in the horizontal direction according to the leftmost key point and the rightmost key point;
according to the cutting width in the vertical direction and the cutting center point in the horizontal direction, cutting the face key point image in a mode that the cutting width is consistent with the cutting height to obtain an initial cutting image;
and performing rotation alignment processing on the initial cutting image based on the face key point image to obtain the face cutting image.
3. The method for repairing a damaged image-based face image according to claim 2, wherein the determining a cropping width in a vertical direction based on the left-eye key point, the right-eye key point, the left-mouth key point and the right-mouth key point comprises:
acquiring the central positions of the key points of the left eye and the right eye as the central positions of the two eyes;
acquiring the central positions of the key points of the left mouth and the right mouth as the central positions of the mouths;
calculating the distance between the center positions of the two eyes and the center position of the mouth to obtain a middle cutting width in the vertical direction, and respectively calculating the middle cutting width, the bottom cutting width and the top cutting width in the vertical direction based on the middle cutting width, the center positions of the two eyes and the center position of the mouth;
and obtaining the cutting width in the vertical direction based on the middle cutting width, the bottom cutting width and the top cutting width.
4. The method for restoring a facial image based on a damaged image according to claim 3, wherein the calculating a distance between the center positions of the two eyes and the center position of the mouth to obtain a middle cropping width in a vertical direction, and calculating the middle cropping width, the bottom cropping width, and the top cropping width in the vertical direction based on the middle cropping width, the center positions of the two eyes, and the center position of the mouth respectively comprises:
calculating the distance between the center positions of the two eyes and the center position of the mouth to obtain the cutting width of the middle part;
according to the middle cutting width, calculating a first preset proportional distance from the center position of the mouth to the vertical direction, and obtaining the bottom cutting width;
and calculating a second preset proportional distance from the center positions of the two eyes to the vertical direction upwards according to the middle cutting width to obtain the top cutting width.
5. The method for repairing facial image based on damaged image as claimed in claim 1, wherein said obtaining a transformation matrix based on the difference between the key point positions of said cut facial image and the key point image of the facial image, and performing affine transformation on said cut facial image according to said transformation matrix to obtain a facial correction image comprises:
respectively acquiring key point positions in the face cutting image and the face key point image to obtain a cutting key point position and a marking key point position;
performing corresponding subtraction processing on the positions of the cutting key points and the positions of the marking key points to obtain the position difference of the key points, and constructing the transformation matrix based on the position difference of the key points;
and performing affine transformation on the cut face image according to the transformation matrix to obtain the face correction image.
6. The method for repairing a facial image based on a damaged image according to claim 1, wherein the performing depth repair on the face-corrected image to obtain a depth repair result, and generating a target repair image based on the depth repair result comprises:
performing depth restoration on the face correction image based on a depth neural network and a generated confrontation network to obtain a depth restoration result;
carrying out inverse transformation processing on the depth restoration result to obtain an inverse transformation image;
and identifying a face region in the preliminary repairing image, and pasting the inverse transformation image back to the face region to obtain the target repairing image.
7. The method for repairing facial image based on damaged image according to any claim 1 to 6, wherein the obtaining of the image to be processed, the detecting of the scratch area of the image to be processed to obtain the scratch area, and the repairing of the scratch area to obtain the preliminary repairing image comprises:
acquiring the image to be processed, and detecting a scratch area of the image to be processed to obtain a scratch area mask;
performing expansion treatment on the mask of the scratch area to obtain the scratch area;
and repairing the scratch area by adopting a preset repairing mode to obtain the primary repairing image.
8. A facial image restoration device based on a damaged image is characterized by comprising:
the device comprises a to-be-processed image acquisition module, a scratch detection module and a repair module, wherein the to-be-processed image acquisition module is used for acquiring an image to be processed, detecting a scratch area of the image to be processed to obtain the scratch area, and repairing the scratch area to obtain a primary repair image;
the face key point detection module is used for detecting face key points of the preliminary restoration image to obtain a face key point image;
the face cutting image generation module is used for cutting and rotationally aligning the face key point image to obtain a face cutting image, wherein the face cutting image comprises face cutting key points;
the correction image generation module is used for obtaining a transformation matrix based on the key point position difference between the human face cutting image and the human face key point image, and carrying out affine transformation on the cutting human face image according to the transformation matrix to obtain a human face correction image;
and the target restoration image generation module is used for performing depth restoration on the face correction image to obtain a depth restoration result and generating a target restoration image based on the depth restoration result.
9. A computer device comprising a memory in which a computer program is stored and a processor that, when executing the computer program, implements a method for repairing a facial image based on a damaged image according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the method for repairing a facial image based on a damaged image according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211189288.2A CN115456906A (en) | 2022-09-28 | 2022-09-28 | Damaged image-based facial image restoration method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211189288.2A CN115456906A (en) | 2022-09-28 | 2022-09-28 | Damaged image-based facial image restoration method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115456906A true CN115456906A (en) | 2022-12-09 |
Family
ID=84306663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211189288.2A Pending CN115456906A (en) | 2022-09-28 | 2022-09-28 | Damaged image-based facial image restoration method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115456906A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117952877A (en) * | 2024-03-26 | 2024-04-30 | 松立控股集团股份有限公司 | Low-quality image correction method based on hierarchical structure modeling |
-
2022
- 2022-09-28 CN CN202211189288.2A patent/CN115456906A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117952877A (en) * | 2024-03-26 | 2024-04-30 | 松立控股集团股份有限公司 | Low-quality image correction method based on hierarchical structure modeling |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11164323B2 (en) | Method for obtaining image tracking points and device and storage medium thereof | |
| US20120134588A1 (en) | Rectification of characters and text as transform invariant low-rank textures | |
| CN108280829A (en) | Welding seam method, computer installation and computer readable storage medium | |
| CN107169489B (en) | Method and apparatus for tilt image correction | |
| WO2021237909A1 (en) | Table restoration method and apparatus, device, and storage medium | |
| CN111259878A (en) | Method and equipment for detecting text | |
| CN102324134A (en) | Valuable document identification method and device | |
| CN111754536B (en) | Image labeling method, device, electronic equipment and storage medium | |
| CN112017231A (en) | Human body weight identification method and device based on monocular camera and storage medium | |
| CN108573251A (en) | Character area localization method and device | |
| CN109741273A (en) | A kind of mobile phone photograph low-quality images automatically process and methods of marking | |
| CN113947768A (en) | Monocular 3D target detection-based data enhancement method and device | |
| CN110502948B (en) | Restoration method and device for folding two-dimensional code image and code scanning equipment | |
| CN115937003A (en) | Image processing method, image processing device, terminal equipment and readable storage medium | |
| CN113228105A (en) | Image processing method and device and electronic equipment | |
| CN111192312A (en) | Depth image acquisition method, device, equipment and medium based on deep learning | |
| CN112950528A (en) | Certificate posture determining method, model training method, device, server and medium | |
| WO2024002396A2 (en) | Vehicle charging port recognition method and related device | |
| CN115456906A (en) | Damaged image-based facial image restoration method, device, equipment and medium | |
| CN114998347B (en) | Semiconductor panel corner positioning method and device | |
| CN108304840B (en) | Image data processing method and device | |
| US20230368324A1 (en) | Method and apparatus of embedding image in video, and method and apparatus of acquiring plane prediction model | |
| CN111640058B (en) | Image fusion processing method and device | |
| WO2022056875A1 (en) | Method and apparatus for segmenting nameplate image, and computer-readable storage medium | |
| CN113870190A (en) | Vertical line detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |