CN111914785B - Method, device and storage medium for improving definition of face image - Google Patents

Method, device and storage medium for improving definition of face image Download PDF

Info

Publication number
CN111914785B
CN111914785B CN202010798214.3A CN202010798214A CN111914785B CN 111914785 B CN111914785 B CN 111914785B CN 202010798214 A CN202010798214 A CN 202010798214A CN 111914785 B CN111914785 B CN 111914785B
Authority
CN
China
Prior art keywords
image
sample
fuzzy
integral
clear
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010798214.3A
Other languages
Chinese (zh)
Other versions
CN111914785A (en
Inventor
万韶华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Xiaomi Pinecone Electronic Co Ltd
Original Assignee
Beijing Xiaomi Pinecone Electronic Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Xiaomi Pinecone Electronic Co Ltd filed Critical Beijing Xiaomi Pinecone Electronic Co Ltd
Priority to CN202010798214.3A priority Critical patent/CN111914785B/en
Publication of CN111914785A publication Critical patent/CN111914785A/en
Application granted granted Critical
Publication of CN111914785B publication Critical patent/CN111914785B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Human Computer Interaction (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computing Systems (AREA)
  • Evolutionary Biology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The present disclosure provides a method, apparatus and storage medium for improving the definition of a face image, the method comprising obtaining a blurred face image; identifying N characteristic areas of the fuzzy face image, determining a fuzzy partial image corresponding to each characteristic area, and obtaining N fuzzy partial images, wherein each fuzzy partial image is a partial image of the fuzzy face image; inputting the blurred face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; and carrying out fusion processing on the clear integral image and the N clear partial images to obtain a fusion integral image. The definition of the important characteristic parts is further improved in a focused manner on the basis of integrally improving the definition, so that more important detail information is embodied.

Description

Method, device and storage medium for improving definition of face image
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a storage medium for improving the definition of a face image.
Background
When a user uses an intelligent terminal to take a picture, the condition that the image is blurred frequently occurs in the shot image due to the limitation of shooting conditions. Such as lens defocus and too far a shooting distance, can cause blurring of the person in the captured image. In a scene of multi-person group photo, the condition that the lens is out of focus to cause the blurring of the person image often occurs, because the lens is focused on only one person, the blurring of the image of other persons occurs.
How to improve the definition of the face image is a problem to be solved.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides a method, an apparatus, and a storage medium for improving the definition of a face image.
According to a first aspect of an embodiment of the present disclosure, there is provided a method for improving sharpness of a face image, including:
acquiring a fuzzy face image; the fuzzy face image comprises a face;
identifying N characteristic areas of the fuzzy face image, determining a fuzzy partial image corresponding to each characteristic area, and obtaining N fuzzy partial images, wherein each fuzzy partial image is a partial image of the fuzzy face image;
Inputting the blurred face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; the definition of the clear integral image is larger than that of the fuzzy face image, and the definition of the clear local image is larger than that of the corresponding fuzzy local image;
performing fusion processing on the clear integral image and the N clear partial images to obtain a fusion integral image;
the N is an integer greater than 0.
In an embodiment, before the obtaining the blurred face image, the method further includes:
acquiring an image to be processed, identifying faces in the image to be processed, determining face area images corresponding to each face, and determining definition of each face area image; and taking the face area image with the definition smaller than the set definition as the fuzzy face image by the fuzzy face image.
In one embodiment, the global map processing model is trained by:
constructing a plurality of integral sample pairs, wherein each integral sample pair comprises a fuzzy integral sample image and a clear integral sample image, and the fuzzy integral sample image is an image obtained by downsampling the clear integral sample image;
And generating an countermeasure network which is used for processing the whole image and is completed by training by using the plurality of whole sample pairs as the whole image processing model.
In one embodiment, the clear monolithic sample images in all monolithic sample pairs are the same size;
the obtaining the blurred face image comprises the following steps:
judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
In one embodiment, the gesture characterization information of the markers in the clear overall images in all the integral sample pairs is the same;
the obtaining the blurred face image comprises the following steps:
judging whether the error between the gesture representation information of the object in the fuzzy face image and the gesture representation information of the object in the clear integral sample image in the integral sample pair is larger than a set error, and correcting the object in the fuzzy face image under the condition that the error is larger than the set error, so that the error between the gesture representation information of the object after correction and the gesture representation information of the object in the clear integral sample image in the integral sample pair is smaller than or equal to the set error.
In one embodiment, the N partial graph processing models are trained by:
identifying N characteristic areas of each fuzzy integral sample image aiming at the fuzzy integral sample image in each integral sample pair, intercepting a fuzzy local sample image corresponding to each characteristic area from the fuzzy integral sample image, intercepting a clear local sample image corresponding to each characteristic area from a clear integral sample image corresponding to the fuzzy integral sample image, and obtaining N local sample pairs, wherein each local sample pair comprises the fuzzy local sample image and the clear local sample image corresponding to the same characteristic area in the same integral sample pair;
and determining N generation countermeasure networks for processing the local images corresponding to the N characteristic areas, training the corresponding generation countermeasure networks for processing the local images by using the local sample pairs corresponding to the same characteristic area, and obtaining N trained generation countermeasure networks for processing the local images as N local image processing models.
In one embodiment, the method further comprises:
determining a fuzzy local sample image corresponding to each characteristic region according to the set size and the position setting rule corresponding to each characteristic region;
Determining a fuzzy local sample image corresponding to each feature area according to the set size and the position setting rule corresponding to each feature area, wherein the fuzzy local sample image comprises the following steps:
determining a feature area coverage area corresponding to each feature area, and determining a fuzzy local sample image containing the corresponding feature area coverage area corresponding to each feature area; the size of the fuzzy local sample image corresponding to each characteristic area is the set size corresponding to the characteristic area, and the position of each characteristic area in the corresponding fuzzy local sample image accords with the corresponding position setting rule;
the location setting rule includes one of the following rules:
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at the central point of the fuzzy local sample image;
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at a set position point of the fuzzy local sample image;
and the boundary line of the setting side of the coverage area of the characteristic area corresponding to the characteristic area is attached to the boundary line of the setting side of the fuzzy local sample image.
In an embodiment, the fusing processing is performed on the clear integral image and the N clear partial images to obtain a fused integral image, which includes:
And inputting the clear integral image and the N clear partial images into a fusion network model to obtain a fusion integral image.
In one embodiment, the converged network model is trained by:
determining a plurality of fusion sample sets, each fusion sample set comprising: a sample integral map for fusion input, N sample partial images for fusion input, and a sample integral map for fusion target; the fusion input sample overall map and the fusion target sample overall map are different images containing the same object, and the N fusion input sample partial images are partial images of different parts in the same object in the input target overall map;
training a neural network using the plurality of fusion sample sets as the fusion network model.
In an embodiment, after training the neural network using the plurality of fusion sample sets, the training method further includes:
and inputting the fusion input sample integral graph in the fusion input sample group into the neural network to obtain an output image output by the neural network, inputting the fusion input sample integral graph and the output image into a face consistency verification network, and updating parameters of the neural network according to an output result of the face consistency verification network.
According to a second aspect of an embodiment of the present disclosure, there is provided an apparatus for improving sharpness of a face image, including:
the first acquisition module is configured to acquire a fuzzy face image; the fuzzy face image comprises a face;
the recognition module is configured to recognize N characteristic areas of the fuzzy face image;
the second acquisition module is configured to determine a fuzzy partial image corresponding to each characteristic region, and obtain N fuzzy partial images, wherein each fuzzy partial image is a partial image of the fuzzy face image;
the processing module is configured to input the fuzzy type face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; the definition of the clear integral image is larger than that of the fuzzy face image, and the definition of the clear local image is larger than that of the corresponding fuzzy local image;
the fusion module is configured to fuse the clear type integral image and the N clear type partial images to obtain a fusion type integral image;
the N is an integer greater than 0.
In one embodiment, the apparatus further comprises:
the first determining module is configured to acquire an image to be processed, identify faces in the image to be processed, determine face area images corresponding to each face, determine definition of each face area image, and take the face area image with definition smaller than the set definition as a fuzzy face image.
In one embodiment, the apparatus comprises:
a first training module configured to train the global graph processing model using the following method;
constructing a plurality of integral sample pairs, wherein each integral sample pair comprises a fuzzy integral sample image and a clear integral sample image, and the fuzzy integral sample image is an image obtained by downsampling the clear integral sample image;
and generating an countermeasure network which is used for processing the whole image and is completed by training by using the plurality of whole sample pairs as the whole image processing model.
In one embodiment, the clear monolithic sample images in all monolithic sample pairs are the same size;
the first acquisition module is further configured to determine a blurred face image using the following method:
judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
In one embodiment, the gesture characterization information of the markers in the clear overall images in all the integral sample pairs is the same;
the first acquisition module is further configured to determine a blurred face image using the following method:
judging whether the gesture representation information of the object in the fuzzy face image is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair, and correcting the object in the fuzzy face image under the condition of different conditions so that the gesture representation information of the object after correction is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair.
In one embodiment, the apparatus comprises:
a second training module configured to train the N partial graph processing models using the following method:
identifying N characteristic areas of each fuzzy integral sample image aiming at the fuzzy integral sample image in each integral sample pair, intercepting a fuzzy local sample image corresponding to each characteristic area from the fuzzy integral sample image, intercepting a clear local sample image corresponding to each characteristic area from a clear integral sample image corresponding to the fuzzy integral sample image, and obtaining N local sample pairs, wherein each local sample pair comprises the fuzzy local sample image and the clear local sample image corresponding to the same characteristic area in the same integral sample pair;
And determining N generation countermeasure networks for processing the local images corresponding to the N characteristic areas, training the corresponding generation countermeasure networks for processing the local images by using the local sample pairs corresponding to the same characteristic area, and obtaining N trained generation countermeasure networks for processing the local images as N local image processing models.
In one embodiment, the apparatus further comprises:
the second determining module is configured to determine a fuzzy local sample image corresponding to each characteristic region according to the set size and the position setting rule corresponding to each characteristic region by using the following method;
determining a feature area coverage area corresponding to each feature area, and determining a fuzzy local sample image containing the corresponding feature area coverage area corresponding to each feature area; the size of the fuzzy local sample image corresponding to each characteristic area is the set size corresponding to the characteristic area, and the position of each characteristic area in the corresponding fuzzy local sample image accords with the corresponding position setting rule;
the location setting rule includes one of the following rules:
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at the central point of the fuzzy local sample image;
The central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at a set position point of the fuzzy local sample image;
and the boundary line of the setting side of the coverage area of the characteristic area corresponding to the characteristic area is attached to the boundary line of the setting side of the fuzzy local sample image.
In an embodiment, the fusion module is configured to perform fusion processing on the clear whole image and the N clear partial images by using the following method to obtain a fused whole image:
and inputting the clear integral image and the N clear partial images into a fusion network model to obtain a fusion integral image.
In one embodiment, the apparatus further comprises:
a third training module configured to train the converged network model using:
determining a plurality of fusion sample sets, each fusion sample set comprising: a sample integral map for fusion input, N sample partial images for fusion input, and a sample integral map for fusion target; the fusion input sample overall map and the fusion target sample overall map are different images containing the same object, and the N fusion input sample partial images are partial images of different parts in the same object in the input target overall map;
Training a neural network using the plurality of fusion sample sets as the fusion network model.
In one embodiment, the apparatus further comprises:
an updating module configured to update the neural network trained using the plurality of fusion-used sample sets using:
and inputting the fusion input sample integral graph in the fusion input sample group into the neural network to obtain an output image output by the neural network, inputting the fusion input sample integral graph and the output image into a face consistency verification network, and updating parameters of the neural network according to an output result of the face consistency verification network.
According to a third aspect of an embodiment of the present disclosure, there is provided an apparatus for improving sharpness of a face image, including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to perform the above method.
According to a fourth aspect of embodiments of the present disclosure, there is provided a non-transitory computer readable storage medium, which when executed by a processor of a mobile terminal, causes the mobile terminal to perform the above-described method.
The technical scheme provided by the embodiment of the disclosure can comprise the following beneficial effects: the method comprises the steps of obtaining a clear face image after processing the blurred face image to improve the definition; and identifying a plurality of feature areas for representing the facial features from the blurred facial image, determining blurred local images corresponding to each feature area, and obtaining a clear local image after carrying out sharpness improvement treatment on each blurred local image. The clear face image and the clear partial images are fused to obtain the fused face image, and the fused face image can further improve the definition of important characteristic parts on the basis of integrally improving the definition, so that more important detail information is reflected, and the capability of carrying the detail information by the image is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a flowchart illustrating a method of improving clarity of a face image, according to an exemplary embodiment;
FIG. 2 is a block diagram illustrating an apparatus for improving clarity of a face image according to an exemplary embodiment;
fig. 3 is a block diagram illustrating an apparatus for improving clarity of a face image according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The embodiment of the disclosure provides a method for improving the definition of a face image. Referring to fig. 1, fig. 1 is a flowchart illustrating a method of improving clarity of a face image according to an exemplary embodiment. As shown in fig. 1, the method includes:
s11, acquiring a fuzzy face image; the blurred face image comprises a face.
Step S12, N feature areas of the fuzzy face image are identified, fuzzy partial images corresponding to each feature area are determined, N fuzzy partial images are obtained, and each fuzzy partial image is a partial image of the fuzzy face image.
Step S13, inputting the blurred face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; the definition of the clear integral image is larger than that of the fuzzy face image, and the definition of the clear local image is larger than that of the corresponding fuzzy local image.
And S14, carrying out fusion processing on the clear integral image and the N clear partial images to obtain a fusion integral image.
Wherein N is an integer greater than 0. For example, N is 1, 2, 3, 4, 5, etc.
The human face in the human face image in the method can be a front image of the human face, or can be a left side image or a right side image of the human face. In one embodiment, the side image of the face is a side image within a set steering angle range. The set steering angle is an angle between a normal line of the front surface of the human face and the photographing direction of the lens. For example: when the steering angle is set to be positive 90 degrees, the face image is the left face. When the steering angle is set to minus 90 degrees, the face image is the right face. For example: the set steering angle range is positive 80 degrees to positive 90 degrees.
In this embodiment, after the processing of improving the definition of the blurred face image, a clear face image is obtained; and identifying a plurality of feature areas for representing the facial features from the blurred facial image, determining blurred local images corresponding to each feature area, and obtaining clear local images after carrying out sharpness improvement treatment on each blurred local image. The clear face image and the clear partial images are fused to obtain the fused face image, and the fused face image can further improve the definition of important characteristic parts on the basis of integrally improving the definition, so that more important detail information is reflected, and the capability of carrying the detail information by the image is improved.
The embodiment of the disclosure provides a method for improving the definition of a face image, which includes the method shown in fig. 1, and further includes, before step S11: acquiring an image to be processed, identifying faces in the image to be processed, determining face area images corresponding to each face, and determining definition of each face area image; and taking the face area image with the definition smaller than the set definition as the fuzzy face image by the fuzzy face image.
In one embodiment, the image to be processed is a multi-person group shot image, with the lens focused on only one person. For example: when a plurality of people are arranged in a row in the front-back order, the lens focuses on the face of the person positioned at the head of the team, and a plurality of face images at the tail of the team are blurred. For example: when a plurality of persons are arranged in a horizontal direction, a plurality of face images on both sides of a team are blurred when a lens is focused on a person located in the middle of the team.
In this embodiment, for a to-be-processed image including a plurality of faces, a blurred face image is cut from the to-be-processed image for subsequent sharpness improvement processing.
The embodiment of the disclosure provides a method for improving the definition of a face image, which includes the method shown in fig. 1, and further includes, before step S11: a process of training the whole graph process model.
The whole graph processing model is trained by:
step 1, constructing a plurality of integral sample pairs, wherein each integral sample pair comprises a fuzzy integral sample image and a clear integral sample image, and the fuzzy integral sample image is an image obtained by downsampling the clear integral sample image.
For example: the blurred whole sample image and the clear whole sample image in one whole sample pair are both front images of the person A. The front image may include only the face region, or may include both the face region and the neck region. The area ratio of the face part in the front face image in the whole image can be different in different integral samples. To ensure training, the area ratio of the human face part in the whole image is within a set ratio range, such as 70-80% ratio, in all integral sample pairs.
And 2, generating an countermeasure network which is used for processing the whole image and is completed by training by using the plurality of whole sample pairs, and taking the generated countermeasure network as a whole image processing model.
Wherein, generating the countermeasure network (Generative Adversarial Networks, GAN for short) is a deep learning model. Generating the antagonism network includes: a Model (generating Model) and a discriminant Model (Discriminative Model) are generated. The generation model and the discrimination model are mutually game-studied, so that the generation countermeasure network completes the study process.
The embodiment of the disclosure provides a method for improving the definition of a face image, which comprises the method shown in fig. 1, and: the clear whole sample images in all whole sample pairs are the same size. In step S11, obtaining a blurred face image includes: judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
For example: the clear whole sample image in the whole sample pair has one of the following dimensions: 640 by 480, 1024 by 768, 1600 by 1200, 2048 by 1536.
The embodiment of the disclosure provides a method for improving the definition of a face image, which comprises the method shown in fig. 1, and the gesture representation information of the objects in the clear integral images in all integral sample pairs is the same.
In step S11, obtaining a blurred face image includes:
judging whether the gesture representation information of the object in the fuzzy face image is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair, and correcting the object in the fuzzy face image under the condition of different conditions so that the gesture representation information of the object after correction is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair.
In an embodiment, the gesture characterization information refers to an included angle between a vertical center line of the face and a vertical edge line of the image. For example: and the included angle between the vertical center line of the human face in the integral sample image and the vertical side line of the integral sample image is 0, which indicates that the posture of the human face is in a right state. In step S11, when the included angle between the vertical center line of the front face in the blurred face image and the vertical edge line of the blurred face image is 0 or smaller than a set angle (for example, 5 degrees), it is determined that the pose of the face is a right state. In step S11, when the included angle between the vertical center line of the front face in the blurred face image and the vertical side line of the blurred face image is greater than a fixed angle (for example, 5 degrees), it is considered that correction needs to be performed on the blurred face image. Correcting the fuzzy face image to make the included angle between the face in the corrected fuzzy face image and the vertical boundary of the fuzzy face image smaller than or equal to the set angle (for example, 5 degrees).
By the correction method, the gestures of the faces in the images are consistent, and the image processing effect can be effectively improved.
The embodiment of the disclosure provides a method for improving the definition of a face image, which includes the method shown in fig. 1, and further includes, before step S11: training N partial graph processing models.
The N partial graph processing models are trained by:
step 1, identifying N characteristic areas of a fuzzy integral sample image aiming at the fuzzy integral sample image in each integral sample pair, intercepting a fuzzy local sample image corresponding to each characteristic area from the fuzzy integral sample image, intercepting a clear local sample image corresponding to each characteristic area from a clear integral sample image corresponding to the fuzzy integral sample image, and obtaining N local sample pairs, wherein each local sample pair comprises the fuzzy local sample image and the clear local sample image corresponding to the same characteristic area in the same integral sample pair.
And 2, determining N generation countermeasure networks for processing the local images corresponding to the N characteristic areas, training the corresponding generation countermeasure networks for processing the local images by using the local sample pairs corresponding to the same characteristic area, and obtaining N trained generation countermeasure networks for processing the local images as N local image processing models.
In one embodiment, the blurred whole sample image in the whole sample pair is a front image of the person a, and the 5 feature areas of the blurred whole sample image are identified as follows: right eye region, left eye region, nose region, mouth region, chin region. And cutting out the fuzzy local sample image corresponding to each characteristic region from the fuzzy integral sample image to obtain 5 fuzzy local sample images. The 5 blurred partial sample images are: a right eye region partial sample image, a left eye region partial sample image, a nose region partial sample image, a mouth region partial sample image, and a chin region partial sample image.
In one embodiment, the blurred whole sample image in the whole sample pair is a side image of the person a, and the N feature areas of the blurred whole sample image are identified as follows: right eye region, nose region, mouth region, ear region. And cutting out the fuzzy local sample image corresponding to each characteristic region from the fuzzy integral sample image to obtain 5 fuzzy local sample images. The 5 blurred partial sample images are: right eye region partial sample image, nose region partial sample image, mouth region partial sample image, ear region partial sample image.
Embodiments of the present disclosure provide a method for improving image sharpness, the method including the method shown in fig. 1, and: the method further comprises the steps of: determining a fuzzy local sample image corresponding to each characteristic region according to the set size and the position setting rule corresponding to each characteristic region;
determining a fuzzy local sample image corresponding to each feature area according to the set size and the position setting rule corresponding to each feature area, wherein the fuzzy local sample image comprises the following steps:
determining a feature area coverage area corresponding to each feature area, and determining a fuzzy local sample image containing the corresponding feature area coverage area corresponding to each feature area; the size of the blurred partial sample image corresponding to each feature area is the set size corresponding to the feature area, and the position of each feature area in the corresponding blurred partial sample image accords with the corresponding position setting rule.
In one embodiment, the size parameters of the overall image include: length 1280 and width 760. The coverage range of the feature area corresponding to the feature area refers to an irregular area comprising local features, and the fuzzy local sample image corresponding to the feature area and containing the coverage range of the corresponding feature area is rectangular. For example: when the characteristic region corresponds to the mouth, the coverage area of the characteristic region corresponding to the mouth characteristic region is an irregular region similar to the shape of the mouth. The blurred partial sample image corresponding to the feature region and containing the coverage of the corresponding feature region is an image corresponding to a rectangular region including the irregular region. For example: the size of the blurred local sample image corresponding to the mouth feature area is 500 and 300.
Similarly, the blurred partial sample image corresponding to the left-eye feature region includes the corresponding feature region coverage of the left eye and is sized to be 128 in length and 38 in width. The blurred partial sample image corresponding to the right eye feature region includes a corresponding feature region coverage for the right eye and is sized to be 128 in length and 38 in width. The blurred partial sample image corresponding to the nose feature area includes a corresponding feature area coverage of the nose and is sized to be 300 in length and 200 in width.
In one embodiment, the location setting rule includes one of the following rules:
and according to a rule I, the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at the central point of the fuzzy local sample image.
And a second rule, wherein the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at a set position point of the fuzzy local sample image.
And thirdly, attaching the boundary line of the setting side of the coverage range of the characteristic region corresponding to the characteristic region to the boundary line of the setting side of the fuzzy local sample image.
For example: when the characteristic region corresponds to the left eye, the coverage area of the characteristic region corresponding to the characteristic region is an irregular region taking the edge of the eye as a boundary.
According to rule one, the center point of the left eye feature area is the center point of the left eye bead, and the center point of the blurred partial sample image is centered on the center point of the left eye bead.
According to rule two, the center point of the left eye feature area is the center point of the left eye bead, the set position point of the blurred partial sample image is a position point located on the longitudinal centerline of the blurred partial sample image, and the distance from the upper edge line is one quarter of the longitudinal centerline. The center point of the left eye feature region is the center point of the left eye bead, and then the center point of the left eye feature region is the above position point of the blurred partial sample image.
According to rule three, the boundary line on the set side of the feature region coverage corresponding to the feature region is the normal line at the left eye corner. The boundary line on the set side of the blurred partial sample image is the boundary line on the left side. The normal at the left corner of the eye in the coverage of the feature region coincides with the boundary line on the left side of the blurred partial sample image.
Embodiments of the present disclosure provide a method for improving image sharpness, the method including the method shown in fig. 1, and: the clear whole sample images in all whole sample pairs are the same size.
In step S11, determining a blurred face image includes:
judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
Embodiments of the present disclosure provide a method for improving image sharpness, the method including the method shown in fig. 1, and: in step S14, fusion processing is performed on the clear integral image and the N clear partial images to obtain a fused integral image, which includes: and inputting the clear integral image and the N clear partial images into a fusion network model to obtain a fusion integral image.
In one embodiment, the method further comprises: training a neural network to obtain the fusion network model. The method specifically comprises the following steps:
determining a plurality of fusion sample sets, each fusion sample set comprising: a sample integral map for fusion input, N sample partial images for fusion input, and a sample integral map for fusion target; the fusion input sample overall map and the fusion target sample overall map are different images containing the same object, and the N fusion input sample partial images are partial images of different parts in the same object;
training the neural network using the plurality of fusion sample sets until training the neural network is successful;
and taking the successfully trained neural network as the fusion network model.
In one embodiment, after training the neural network using the plurality of fusion sample sets, the method further comprises: and inputting the fusion input sample integral graph in the fusion input sample group into the neural network to obtain an output image output by the neural network, inputting the fusion input sample integral graph and the output image into a face consistency verification network, and updating parameters of the neural network according to an output result of the face consistency verification network. Therefore, the face consistency verification network ensures the face consistency processing effect of the neural network, ensures that the appearance of the face in the image after the definition is improved is kept to be higher consistent with that in the image before the definition is improved, and prevents serious appearance change.
The following is a detailed description of specific examples.
Example 1
In one example, the face feature regions are combined and fused by using the facial feature regions in the front face.
A first network is determined, the first network being a first GAN. This first GAN is used to handle the generation of the overall image against the network. 100 whole sample pairs were constructed. The target object contained in the images in all the integral sample pairs is a frontal face. Each integral sample pair contains both blurred and sharp face images of the same person. Faces in different integral sample pairs belong to different people. All images in the integral sample pair are sized to be length 1280 and width 760. After training the first GAN successfully using the plurality of whole sample pairs, using the first GAN successfully trained as a whole graph processing model. The overall face map processing model can process the blurred overall face map into a clear overall face map.
The characteristic areas are determined to be left eye, right eye, nose and mouth respectively.
A second network is determined, the first network being a second GAN, and 100 first local pairs of samples corresponding to the left-eye feature region are constructed. Each first partial sample pair contains a blurred left-eye image and a sharp left-eye image of the same person. The left eye of the different first partial sample pair belongs to a different person. All images in the first local sample pair are of length 128 and width 38. After training the second GAN using 100 pairs of first local patterns, the second GAN that was successfully trained is used as the first local graph processing model. This partial graph processing model can process a blurred left eye image into a sharp left eye image.
A third network is determined, which is a third GAN, and 100 second local sample pairs corresponding to the right-eye feature region are constructed. Each second partial sample pair contains a blurred right-eye image and a sharp right-eye image of the same person. The right eye of the second, different local sample pair belongs to a different person. All images in the second local sample pair are of length 128 and width 38. After training the third GAN using 100 second local sample pairs, using the trained third GAN as the second local graph processing model. This partial graph processing model can process a blurred right-eye image into a sharp right-eye image.
A fourth network is determined, which is a fourth GAN, and 100 third local sample pairs corresponding to the nose feature region are constructed. Each third partial sample pair contains a blurred nose image and a sharp nose image of the same person. The nose in the third, different local sample pair belongs to a different person. All images in the third local sample pair are sized to be 300 in length and 200 in width. After training the fourth GAN using 100 third local sample pairs, using the successfully trained fourth GAN as the third local graph processing model. This partial graph processing model can process a blurred nose image into a clear nose image.
A fifth network is determined, the fifth network being a fifth GAN, and 100 fourth local sample pairs corresponding to the mouth feature region are constructed. Each fourth partial sample pair contains a blurred mouth image and a sharp mouth image of the same person. The mouths of the fourth, different local sample pair belong to different people. All images in the fourth local sample pair are sized to be 300 in length and 200 in width. After training the fifth GAN using 100 pairs of fourth local patterns, the fifth GAN that was successfully trained was used as the fourth local graph processing model. This partial graph processing model may process the blurred mouth image into a sharp mouth image.
A sixth network is determined, the sixth network being a BP neural network. 100 fusion sample sets were constructed, each comprising: a sample overall diagram for fusion input, N sample partial images for fusion input and a sample overall diagram for fusion target. The input sample whole graph and the fusion target sample in each fusion sample group are images of the same face, and the expression of the same face in the input sample whole graph and the fusion target sample is slightly different. The N fused input sample partial images are partial images of different parts of the human face in the input target whole image.
And obtaining a multi-person group photo shot by a user through a mobile phone, and cutting out an image which comprises a blurred face image and has a size of length 1280 and width 760 from the multi-person group photo as a target image.
4 feature areas of left eye, right eye, nose and mouth are identified from the target image, and corresponding fuzzy partial images are determined according to the 4 feature areas, specifically:
a first blurred partial image corresponding to the left eye having a dimension of length 128 and width 38;
a second blurred partial image corresponding to the right eye having a dimension of length 128 and width 38;
a third blurred partial image corresponding to a size of the nose of length 300 and width 200;
A third blurred partial image corresponding to a size of the mouth of length 500 and width 300.
And inputting the target image into a trained first network to obtain a clear integral image.
And inputting the first fuzzy partial image into a trained second network to obtain a first clear partial image.
And inputting the second fuzzy partial image into a trained third network to obtain a second clear partial image.
And inputting the third blurred partial image into a trained fourth network to obtain a third clear partial image.
And inputting the fourth fuzzy partial image into a trained fifth network to obtain a fourth clear partial image.
And inputting the clear integral image, the first clear partial image, the second clear partial image, the third clear partial image and the fourth clear partial image into a sixth network to obtain the fused clear face image.
The embodiment of the disclosure provides a device for improving the definition of a face image. Referring to fig. 2, fig. 2 is a block diagram illustrating an apparatus for improving clarity of a face image according to an exemplary embodiment. As shown in fig. 2, the apparatus includes:
a first acquisition module 201 configured to acquire a blurred face image; the fuzzy face image comprises a face;
An identification module 202 configured to identify N feature areas of the blurred face image;
a second obtaining module 203, configured to determine a blurred local image corresponding to each feature area, and obtain N blurred local images, where each blurred local image is a partial image of the blurred face image;
a processing module 204 configured to input the blurred face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; the definition of the clear integral image is larger than that of the fuzzy face image, and the definition of the clear local image is larger than that of the corresponding fuzzy local image;
the fusion module 205 is configured to perform fusion processing on the clear integral image and the N clear partial images to obtain a fusion integral image;
the N is an integer greater than 0.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: further comprises:
the first determining module is configured to acquire an image to be processed, identify faces in the image to be processed, determine face area images corresponding to each face, determine definition of each face area image, and take the face area image with definition smaller than the set definition as a fuzzy face image.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: further comprises:
a first training module configured to train the global graph processing model using the following method;
constructing a plurality of integral sample pairs, wherein each integral sample pair comprises a fuzzy integral sample image and a clear integral sample image, and the fuzzy integral sample image is an image obtained by downsampling the clear integral sample image;
and generating an countermeasure network which is used for processing the whole image and is completed by training by using the plurality of whole sample pairs as the whole image processing model.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: the clear integral sample images in all integral sample pairs are the same in size;
the first acquisition module 201 is further configured to determine a blurred face image using the following method:
judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: the gesture representation information of the markers in the clear integral images in all integral sample pairs is the same;
the first acquisition module 201 is further configured to determine a blurred face image using the following method:
judging whether the gesture representation information of the object in the fuzzy face image is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair, and correcting the object in the fuzzy face image under the condition of different conditions so that the gesture representation information of the object after correction is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: further comprises:
a second training module configured to train the N partial graph processing models using the following method:
identifying N characteristic areas of each fuzzy integral sample image aiming at the fuzzy integral sample image in each integral sample pair, intercepting a fuzzy local sample image corresponding to each characteristic area from the fuzzy integral sample image, intercepting a clear local sample image corresponding to each characteristic area from a clear integral sample image corresponding to the fuzzy integral sample image, and obtaining N local sample pairs, wherein each local sample pair comprises the fuzzy local sample image and the clear local sample image corresponding to the same characteristic area in the same integral sample pair;
And determining N generation countermeasure networks for processing the local images corresponding to the N characteristic areas, training the corresponding generation countermeasure networks for processing the local images by using the local sample pairs corresponding to the same characteristic area, and obtaining N trained generation countermeasure networks for processing the local images as N local image processing models.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: further comprises:
the second determining module is configured to determine a fuzzy local sample image corresponding to each characteristic region according to the set size and the position setting rule corresponding to each characteristic region by using the following method;
determining a feature area coverage area corresponding to each feature area, and determining a fuzzy local sample image containing the corresponding feature area coverage area corresponding to each feature area; the size of the fuzzy local sample image corresponding to each characteristic area is the set size corresponding to the characteristic area, and the position of each characteristic area in the corresponding fuzzy local sample image accords with the corresponding position setting rule;
the location setting rule includes one of the following rules:
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at the central point of the fuzzy local sample image;
The central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at a set position point of the fuzzy local sample image;
and the boundary line of the setting side of the coverage area of the characteristic area corresponding to the characteristic area is attached to the boundary line of the setting side of the fuzzy local sample image.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and:
the fusion module 205 is configured to perform fusion processing on the clear integral image and the N clear partial images by using the following method to obtain a fused integral image:
and inputting the clear integral image and the N clear partial images into a fusion network model to obtain a fusion integral image.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: further comprises:
a third training module configured to train the converged network model using:
determining a plurality of fusion sample sets, each fusion sample set comprising: a sample integral map for fusion input, N sample partial images for fusion input, and a sample integral map for fusion target; the fusion input sample overall map and the fusion target sample overall map are different images containing the same object, and the N fusion input sample partial images are partial images of different parts in the same object in the input target overall map;
Training a neural network using the plurality of fusion sample sets as the fusion network model.
The embodiment of the disclosure provides a device for improving the definition of a face image, which comprises a device shown in fig. 2, and: further comprises:
an updating module configured to update the neural network trained using the plurality of fusion-used sample sets using:
and inputting the fusion input sample integral graph in the fusion input sample group into the neural network to obtain an output image output by the neural network, inputting the fusion input sample integral graph and the output image into a face consistency verification network, and updating parameters of the neural network according to an output result of the face consistency verification network.
An embodiment of the present disclosure provides a device for improving clarity of a face image, including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured as the method described above.
Embodiments of the present disclosure provide a non-transitory computer-readable storage medium that, when executed by a processor of a mobile terminal, enables the mobile terminal to perform the above-described method.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
Fig. 3 is a block diagram illustrating an example embodiment of a method 300 for improving clarity of a face image. For example, apparatus 300 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.
Referring to fig. 3, apparatus 300 may include one or more of the following components: a processing component 302, a memory 304, a power supply component 306, a multimedia component 308, an audio component 310, an input/output (I/O) interface 312, a sensor component 314, and a communication component 316.
The processing component 302 generally controls overall operation of the apparatus 300, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 302 can include one or more modules that facilitate interactions between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate interaction between the multimedia component 308 and the processing component 302.
Memory 304 is configured to store various types of data to support operations at device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phonebook data, messages, pictures, videos, and the like. The memory 304 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 306 provides power to the various components of the device 300. The power supply components 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 300.
The multimedia component 308 includes a screen between the device 300 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front-facing camera and/or a rear-facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 300 is in an operational mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 310 is configured to output and/or input audio signals. For example, the audio component 310 includes a Microphone (MIC) configured to receive external audio signals when the device 300 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 304 or transmitted via the communication component 316. In some embodiments, audio component 310 further comprises a speaker for outputting audio signals.
The I/O interface 312 provides an interface between the processing component 302 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 314 includes one or more sensors for providing status assessment of various aspects of the apparatus 300. For example, the sensor assembly 314 may detect the on/off state of the device 300, the relative positioning of the components, such as the display and keypad of the apparatus 300, the sensor assembly 314 may also detect a change in position of the apparatus 300 or one component of the apparatus 300, the presence or absence of user contact with the apparatus 300, the orientation or acceleration/deceleration of the apparatus 300, and a change in temperature of the apparatus 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 316 is configured to facilitate communication between the apparatus 300 and other devices, either wired or wireless. The device 300 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 300 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 304, including instructions executable by processor 320 of apparatus 300 to perform the above-described method. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (22)

1. A method for improving clarity of a face image, comprising:
acquiring a fuzzy face image; the fuzzy face image comprises a face;
identifying N characteristic areas of the fuzzy face image, determining a fuzzy partial image corresponding to each characteristic area, and obtaining N fuzzy partial images, wherein each fuzzy partial image is a partial image of the fuzzy face image;
Inputting the blurred face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; the definition of the clear integral image is larger than that of the fuzzy face image, and the definition of the clear local image is larger than that of the corresponding fuzzy local image;
performing fusion processing on the clear integral image and the N clear partial images to obtain a fusion integral image;
the N is an integer greater than 0.
2. The method of claim 1, wherein,
before the obtaining of the blurred face image, the method further comprises the following steps:
acquiring an image to be processed, identifying faces in the image to be processed, determining face area images corresponding to each face, and determining definition of each face area image; and taking the face area image with the definition smaller than the set definition as the fuzzy face image by the fuzzy face image.
3. The method of claim 1, wherein,
the whole graph processing model is trained by:
constructing a plurality of integral sample pairs, wherein each integral sample pair comprises a fuzzy integral sample image and a clear integral sample image, and the fuzzy integral sample image is an image obtained by downsampling the clear integral sample image;
And generating an countermeasure network which is used for processing the whole image and is completed by training by using the plurality of whole sample pairs as the whole image processing model.
4. The method of claim 3, wherein,
the clear integral sample images in all integral sample pairs are the same in size;
the obtaining the blurred face image comprises the following steps:
judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
5. The method of claim 3, wherein,
the gesture representation information of the markers in the clear integral images in all integral sample pairs is the same;
the obtaining the blurred face image comprises the following steps:
judging whether the error between the gesture representation information of the object in the fuzzy face image and the gesture representation information of the object in the clear integral sample image in the integral sample pair is larger than a set error, and correcting the object in the fuzzy face image under the condition that the error is larger than the set error, so that the error between the gesture representation information of the object after correction and the gesture representation information of the object in the clear integral sample image in the integral sample pair is smaller than or equal to the set error.
6. The method according to claim 3 to 5,
the N partial graph processing models are trained by:
identifying N characteristic areas of each fuzzy integral sample image aiming at the fuzzy integral sample image in each integral sample pair, intercepting a fuzzy local sample image corresponding to each characteristic area from the fuzzy integral sample image, intercepting a clear local sample image corresponding to each characteristic area from a clear integral sample image corresponding to the fuzzy integral sample image, and obtaining N local sample pairs, wherein each local sample pair comprises the fuzzy local sample image and the clear local sample image corresponding to the same characteristic area in the same integral sample pair;
and determining N generation countermeasure networks for processing the local images corresponding to the N characteristic areas, training the corresponding generation countermeasure networks for processing the local images by using the local sample pairs corresponding to the same characteristic area, and obtaining N trained generation countermeasure networks for processing the local images as N local image processing models.
7. The method according to claim 3 to 5,
The method further comprises the steps of:
determining a fuzzy local sample image corresponding to each characteristic region according to the set size and the position setting rule corresponding to each characteristic region;
determining a fuzzy local sample image corresponding to each feature area according to the set size and the position setting rule corresponding to each feature area, wherein the fuzzy local sample image comprises the following steps:
determining a feature area coverage area corresponding to each feature area, and determining a fuzzy local sample image containing the corresponding feature area coverage area corresponding to each feature area; the size of the fuzzy local sample image corresponding to each characteristic area is the set size corresponding to the characteristic area, and the position of each characteristic area in the corresponding fuzzy local sample image accords with the corresponding position setting rule;
the location setting rule includes one of the following rules:
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at the central point of the fuzzy local sample image;
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at a set position point of the fuzzy local sample image;
and the boundary line of the setting side of the coverage area of the characteristic area corresponding to the characteristic area is attached to the boundary line of the setting side of the fuzzy local sample image.
8. The method of claim 1, wherein,
the fusion processing is carried out on the clear type integral image and the N clear type local images to obtain a fusion type integral image, which comprises the following steps:
and inputting the clear integral image and the N clear partial images into a fusion network model to obtain a fusion integral image.
9. The method of claim 8, wherein,
the converged network model is trained by:
determining a plurality of fusion sample sets, each fusion sample set comprising: a sample integral map for fusion input, N sample partial images for fusion input, and a sample integral map for fusion target; the fusion input sample overall map and the fusion target sample overall map are different images containing the same object, and the N fusion input sample partial images are partial images of different parts in the same object in the input target overall map;
training a neural network using the plurality of fusion sample sets as the fusion network model.
10. The method of claim 9, wherein,
after training the neural network using the plurality of fusion sample sets, the method further comprises:
And inputting the fusion input sample integral graph in the fusion input sample group into the neural network to obtain an output image output by the neural network, inputting the fusion input sample integral graph and the output image into a face consistency verification network, and updating parameters of the neural network according to an output result of the face consistency verification network.
11. A device for improving the sharpness of a face image, comprising:
the first acquisition module is configured to acquire a fuzzy face image; the fuzzy face image comprises a face;
the recognition module is configured to recognize N characteristic areas of the fuzzy face image;
the second acquisition module is configured to determine a fuzzy partial image corresponding to each characteristic region, and obtain N fuzzy partial images, wherein each fuzzy partial image is a partial image of the fuzzy face image;
the processing module is configured to input the fuzzy type face image into an overall image processing model to obtain a clear overall image; inputting each fuzzy partial image into a corresponding partial image processing model respectively to obtain N clear partial images; the definition of the clear integral image is larger than that of the fuzzy face image, and the definition of the clear local image is larger than that of the corresponding fuzzy local image;
The fusion module is configured to fuse the clear type integral image and the N clear type partial images to obtain a fusion type integral image;
the N is an integer greater than 0.
12. The apparatus of claim 11, wherein the device comprises a plurality of sensors,
the apparatus further comprises:
the first determining module is configured to acquire an image to be processed, identify faces in the image to be processed, determine face area images corresponding to each face, determine definition of each face area image, and take the face area image with definition smaller than the set definition as a fuzzy face image.
13. The apparatus of claim 11, wherein the device comprises a plurality of sensors,
the device comprises:
a first training module configured to train the global graph processing model using the following method;
constructing a plurality of integral sample pairs, wherein each integral sample pair comprises a fuzzy integral sample image and a clear integral sample image, and the fuzzy integral sample image is an image obtained by downsampling the clear integral sample image;
and generating an countermeasure network which is used for processing the whole image and is completed by training by using the plurality of whole sample pairs as the whole image processing model.
14. The apparatus of claim 13, wherein the device comprises a plurality of sensors,
the clear integral sample images in all integral sample pairs are the same in size;
the first acquisition module is further configured to determine a blurred face image using the following method:
judging whether the size of the fuzzy face image is larger than that of the clear integral image in the integral sample pair, and if so, cutting the fuzzy face image, and taking the cut image as an updated fuzzy face image.
15. The apparatus of claim 13, wherein the device comprises a plurality of sensors,
the gesture representation information of the markers in the clear integral images in all integral sample pairs is the same;
the first acquisition module is further configured to determine a blurred face image using the following method:
judging whether the gesture representation information of the object in the fuzzy face image is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair, and correcting the object in the fuzzy face image under the condition of different conditions so that the gesture representation information of the object after correction is the same as the gesture representation information of the object in the clear integral sample image in the integral sample pair.
16. The apparatus according to any one of claim 13 to 15, wherein,
the device comprises:
a second training module configured to train the N partial graph processing models using the following method:
identifying N characteristic areas of each fuzzy integral sample image aiming at the fuzzy integral sample image in each integral sample pair, intercepting a fuzzy local sample image corresponding to each characteristic area from the fuzzy integral sample image, intercepting a clear local sample image corresponding to each characteristic area from a clear integral sample image corresponding to the fuzzy integral sample image, and obtaining N local sample pairs, wherein each local sample pair comprises the fuzzy local sample image and the clear local sample image corresponding to the same characteristic area in the same integral sample pair;
and determining N generation countermeasure networks for processing the local images corresponding to the N characteristic areas, training the corresponding generation countermeasure networks for processing the local images by using the local sample pairs corresponding to the same characteristic area, and obtaining N trained generation countermeasure networks for processing the local images as N local image processing models.
17. The apparatus according to any one of claim 13 to 15, wherein,
the apparatus further comprises:
the second determining module is configured to determine a fuzzy local sample image corresponding to each characteristic region according to the set size and the position setting rule corresponding to each characteristic region by using the following method;
determining a feature area coverage area corresponding to each feature area, and determining a fuzzy local sample image containing the corresponding feature area coverage area corresponding to each feature area; the size of the fuzzy local sample image corresponding to each characteristic area is the set size corresponding to the characteristic area, and the position of each characteristic area in the corresponding fuzzy local sample image accords with the corresponding position setting rule;
the location setting rule includes one of the following rules:
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at the central point of the fuzzy local sample image;
the central point of the coverage range of the characteristic area corresponding to the characteristic area is positioned at a set position point of the fuzzy local sample image;
and the boundary line of the setting side of the coverage area of the characteristic area corresponding to the characteristic area is attached to the boundary line of the setting side of the fuzzy local sample image.
18. The apparatus of claim 17, wherein the device comprises a plurality of sensors,
the fusion module is configured to perform fusion processing on the clear integral image and the N clear partial images by using the following method to obtain a fused integral image:
and inputting the clear integral image and the N clear partial images into a fusion network model to obtain a fusion integral image.
19. The apparatus of claim 18, wherein the device comprises a plurality of sensors,
the apparatus further comprises:
a third training module configured to train the converged network model using:
determining a plurality of fusion sample sets, each fusion sample set comprising: a sample integral map for fusion input, N sample partial images for fusion input, and a sample integral map for fusion target; the fusion input sample overall map and the fusion target sample overall map are different images containing the same object, and the N fusion input sample partial images are partial images of different parts in the same object in the input target overall map;
training a neural network using the plurality of fusion sample sets as the fusion network model.
20. The apparatus of claim 19, wherein the device comprises a plurality of sensors,
the apparatus further comprises:
an updating module configured to update the neural network trained using the plurality of fusion-used sample sets using:
and inputting the fusion input sample integral graph in the fusion input sample group into the neural network to obtain an output image output by the neural network, inputting the fusion input sample integral graph and the output image into a face consistency verification network, and updating parameters of the neural network according to an output result of the face consistency verification network.
21. A device for improving the sharpness of a face image, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to perform the method of any of claims 1 to 10.
22. A non-transitory computer readable storage medium, which when executed by a processor of a mobile terminal, causes the mobile terminal to perform the method of any of claims 1 to 10.
CN202010798214.3A 2020-08-10 2020-08-10 Method, device and storage medium for improving definition of face image Active CN111914785B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010798214.3A CN111914785B (en) 2020-08-10 2020-08-10 Method, device and storage medium for improving definition of face image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010798214.3A CN111914785B (en) 2020-08-10 2020-08-10 Method, device and storage medium for improving definition of face image

Publications (2)

Publication Number Publication Date
CN111914785A CN111914785A (en) 2020-11-10
CN111914785B true CN111914785B (en) 2023-12-05

Family

ID=73283705

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010798214.3A Active CN111914785B (en) 2020-08-10 2020-08-10 Method, device and storage medium for improving definition of face image

Country Status (1)

Country Link
CN (1) CN111914785B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112801946B (en) * 2021-01-13 2024-03-29 福建星网天合智能科技有限公司 Method, device, equipment and medium for calculating face definition
CN113344832A (en) * 2021-05-28 2021-09-03 杭州睿胜软件有限公司 Image processing method and device, electronic equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101609549A (en) * 2009-07-24 2009-12-23 河海大学常州校区 The multi-scale geometric analysis super-resolution processing method of video blurred image
CN102542545A (en) * 2010-12-24 2012-07-04 方正国际软件(北京)有限公司 Multi-focal length photo fusion method and system and photographing device
WO2019214381A1 (en) * 2018-05-09 2019-11-14 腾讯科技（深圳）有限公司 Video deblurring method and apparatus, and storage medium and electronic apparatus
WO2019218621A1 (en) * 2018-05-18 2019-11-21 北京市商汤科技开发有限公司 Detection method for living being, device, electronic apparatus, and storage medium
CN111368685A (en) * 2020-02-27 2020-07-03 北京字节跳动网络技术有限公司 Key point identification method and device, readable medium and electronic equipment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109325933B (en) * 2017-07-28 2022-06-21 阿里巴巴集团控股有限公司 Method and device for recognizing copied image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101609549A (en) * 2009-07-24 2009-12-23 河海大学常州校区 The multi-scale geometric analysis super-resolution processing method of video blurred image
CN102542545A (en) * 2010-12-24 2012-07-04 方正国际软件(北京)有限公司 Multi-focal length photo fusion method and system and photographing device
WO2019214381A1 (en) * 2018-05-09 2019-11-14 腾讯科技（深圳）有限公司 Video deblurring method and apparatus, and storage medium and electronic apparatus
WO2019218621A1 (en) * 2018-05-18 2019-11-21 北京市商汤科技开发有限公司 Detection method for living being, device, electronic apparatus, and storage medium
CN111368685A (en) * 2020-02-27 2020-07-03 北京字节跳动网络技术有限公司 Key point identification method and device, readable medium and electronic equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于编解码网络的多姿态人脸图像正面化方法;徐海月;姚乃明;彭晓兰;陈辉;王宏安;;中国科学:信息科学(04);全文 *

Also Published As

Publication number Publication date
CN111914785A (en) 2020-11-10

Similar Documents

Publication Publication Date Title
CN109784255B (en) Neural network training method and device and recognition method and device
CN105488527B (en) Image classification method and device
EP3383022A1 (en) Method and camera device for processing image
KR101694643B1 (en) Method, apparatus, device, program, and recording medium for image segmentation
CN106331504B (en) Shooting method and device
CN111553864B (en) Image restoration method and device, electronic equipment and storage medium
CN105554389B (en) Shooting method and device
CN111105454B (en) Method, device and medium for obtaining positioning information
CN107944367B (en) Face key point detection method and device
CN110969120B (en) Image processing method and device, electronic equipment and readable storage medium
CN110909654A (en) Training image generation method and device, electronic equipment and storage medium
CN109325908B (en) Image processing method and device, electronic equipment and storage medium
CN110490164B (en) Method, device, equipment and medium for generating virtual expression
EP3945494A1 (en) Video processing method, apparatus and storage medium
CN106503682B (en) Method and device for positioning key points in video data
US11252341B2 (en) Method and device for shooting image, and storage medium
CN111914785B (en) Method, device and storage medium for improving definition of face image
US9665925B2 (en) Method and terminal device for retargeting images
CN109167921B (en) Shooting method, shooting device, shooting terminal and storage medium
CN111507131A (en) Living body detection method and apparatus, electronic device, and storage medium
CN115601316A (en) Image processing method, image processing device, electronic equipment and computer readable storage medium
CN113315904B (en) Shooting method, shooting device and storage medium
CN114511754A (en) Sample image generation method and device, electronic equipment and storage medium
CN114445298A (en) Image processing method and device, electronic equipment and storage medium
CN111985280B (en) Image processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant