CN115798014A

CN115798014A - Image processing method and device

Info

Publication number: CN115798014A
Application number: CN202211585281.2A
Authority: CN
Inventors: 冀盛
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2022-12-09
Filing date: 2022-12-09
Publication date: 2023-03-14

Abstract

The application discloses an image processing method and an image processing device, and belongs to the technical field of communication. The method comprises the following steps: determining at least one face image area in a first face image; under the condition that the face image in the first face image area has face distortion, generating a target face image based on the first face image and the second face image; the face image in the first face image area and the second face image are face images including the same target object, and the second face image does not have face distortion.

Description

Image processing method and device

Technical Field

The present application belongs to the field of image processing technology, and in particular, relates to an image processing method and apparatus.

Background

With the popularization of electronic devices, the appearance of various electronic devices brings great convenience to the life of users, the shooting function is one of important functions in electronic devices such as mobile phones, tablet computers or electronic readers, and the users can use the shooting function of the electronic devices to carry out self-shooting, portrait shooting, group photo and the like.

Generally, when an image including a frontal face of one or more users is shot by using an electronic device, facial expression abnormality may occur to any user of the one or more users, and the facial expression abnormality may include squinting, closing eyes, sipping mouth, squinting and the like, so that in an image finally collected by the electronic device, a situation that the facial expression of the user is not good may exist, and the quality of the shot image is poor. In the related art, in order to obtain an image with a good effect, a user generally needs to perform a beautification process on the image by using a cropping software, and for example, when processing a squinting image, the user needs to introduce the image into the cropping software and then perform pixel stretching on an eye area in the image by using a cropping tool of the cropping software to crop the eye area. Thus, the processing procedure of the image is complicated and the operation difficulty is high.

Disclosure of Invention

An object of the embodiments of the present application is to provide an image processing method and an image processing apparatus, which can solve the problems of complicated steps and high operation difficulty in the image processing process.

In a first aspect, an embodiment of the present application provides an image processing method, including: determining at least one face image region in a first face image; under the condition that the face image in the first face image area has face distortion, generating a target face image based on the first face image and the second face image; the face image in the first face image region and the second face image region are face images including the same target object, and the second face image does not have face distortion.

In a second aspect, an embodiment of the present application provides an image processing apparatus, including: a determination module and a generation module, wherein: the determining module is used for determining at least one face image area in the first face image; the generating module is used for generating a target face image based on the first face image and the second face image under the condition that the face image in the first face image area has face distortion; the face image in the first face image region and the second face image region are face images including the same target object, and the second face image does not have face distortion.

In a third aspect, embodiments of the present application provide an electronic device, which includes a processor and a memory, where the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

In a fourth aspect, embodiments of the present application provide a readable storage medium, on which a program or instructions are stored, which when executed by a processor, implement the steps of the method according to the first aspect.

In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the method according to the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method according to the first aspect.

In the embodiment of the application, the image processing device determines at least one face image area in a first face image, and generates a target face image based on the first face image and a second face image under the condition that the face image in the first face image area has face distortion; the face image in the first face image region and the second face image region are face images including the same target object, and the second face image has no face distortion. By the method, the image processing device can generate the target face image based on the second face image and the first face image under the condition that the face area in the current first face image has face distortion, and because the images in the face areas in the second face image and the first face image are the face images comprising the same object and the second face image does not have face distortion, the target face image without face distortion is generated by utilizing the historical face image with better quality of the target object and the face image with face distortion of the target object, so that the face image with face distortion in the face image can be accurately corrected, the face expression is greatly improved, and the image with higher quality is obtained.

Drawings

Fig. 1 is a flowchart of an image processing method according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a data set provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a neural network model training process provided in an embodiment of the present application;

fig. 4 is a second flowchart of an image processing method according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 7 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present disclosure.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in other sequences than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of objects, e.g., a first object can be one or more. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/", and generally means that the former and latter related objects are in an "or" relationship.

The image processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings by using specific embodiments and application scenarios thereof.

With the popularization of mobile terminals, especially intelligent electronic devices, the use of portrait beautifying and enhancing technology in photographing is increasing. Users are increasingly demanding for use of electronic devices such as cell phones for self-photography, portrait photography, group photography, and the like.

In the related art, several frames captured at the time of photographing are generally used for quality enhancement. Specifically, in the existing scheme, when an image is shot, the image quality is improved by adopting modes such as optimizing the light incoming amount of a sensor and improving the image processing algorithm effect of a camera, however, the existing scheme has the following defects:

1) The processing of images shot under shaking and dim light is poor, the human face is easy to blur, and dim light noise is obvious; 2) Under the condition that the face accounts for a small amount in the group photo, the problem of unclear face exists due to limited pixel resolution; 3) The problems of eye closure, poor expression and the like in the shot image cannot be avoided.

In the image processing method provided by the embodiment of the application, the image processing device can generate the natural facial image with the expression of the target object by using the natural photo with the historical expression of the target object and the current photo to be processed of the target object, so that the problem of poor facial expression in group photo is avoided, and the facial image of each object can be used as training data to obtain the image enhancement network model of a specially-assigned person, so that the enhancement can be performed on a single person in group photo, the problems of image blurring, image flaws and the like are solved, the image enhancement network model of the specially-assigned person is used, the details and noise expression of the face are improved, the original skin detail characteristics can be effectively reserved, and the side effects of false details and removal of characteristic nevi as noise and the like are not easily introduced.

Fig. 1 is a flowchart of an image processing method provided in an embodiment of the present application, and as shown in fig. 1, the image processing method provided in the embodiment of the present application may include the following steps 201 and 202:

step 201: the image processing means determines at least one face image region in the first face image.

Optionally, in this embodiment of the application, the first face image may be a single-person photograph or a multi-person group photograph.

Optionally, in this embodiment of the application, the first face image may be an image stored locally in the electronic device, for example, the first face image is an image in a gallery of the electronic device, or the first face image is a face image received from a recipient, for example, the first face image is an image received from an electronic device corresponding to contact 1.

Alternatively, in this embodiment, the first face image may be an image selected by a user or an image automatically acquired by an image processing apparatus. For example, the user may select a photo to be processed from the gallery to beautify and enhance the photo.

Optionally, in this embodiment of the application, the image processing apparatus may perform face detection on the first face image to determine at least one face image region in the first face image.

For example, in the case that the first face image is a single person photo of the object a, the image processing apparatus may perform face detection on the single person photo by using a preset face detection algorithm, and determine a face image area in the single person photo, where the face image area is an image area corresponding to the face of the object a.

For example, in a case where the first face image is a group photo of the object a, the object B, and the object C, the image processing apparatus may perform face detection on the group photo by using a preset face detection algorithm, and determine three face image regions in the group photo, where the three face image regions are image regions corresponding to faces of the object a, the object B, and the object C, respectively.

Step 202: when the face image in the first face image region has face distortion, the image processing apparatus generates a target face image based on the first face image and the second face image.

The face image in the first face image region and the second face image region are face images including the same target object, and the second face image has no face distortion.

That is, the facial image in the first facial image region and the second facial image are facial images captured for the same target object.

Optionally, in this embodiment of the application, the first facial image region may be one or more facial image regions in the at least one facial image region.

Illustratively, in a case where the at least one face image region includes one face image region, the first face region is the one face image region.

For example, in a case where the at least one face image region includes a plurality of face image regions, the first face region is at least one of the plurality of face image regions.

Optionally, in this embodiment of the application, the first facial image region may be a facial image region selected by a user from the at least one facial image region, or the first facial image may be a facial image region automatically determined by the image processing apparatus from the at least one facial image region.

Illustratively, in the case that a first face image is taken as a group photo of an object a, an object B and an object C, the image processing device displays the group photo on an interface, performs face detection on the group photo, determines image areas corresponding to faces of the object a, the object B and the object C in the group photo, then displays rectangular frames to frame and select image areas corresponding to faces of the object a, the object B and the object C in the group photo respectively, from which a user can select one or more image areas to be processed, and after the user clicks the image area corresponding to the face of the object a, the image processing device determines the image area corresponding to the face of the object a as the first face image area to process a face image of the face area corresponding to the object a in the following.

For example, in combination with the above-mentioned embodiments, when the image processing apparatus performs face detection on the group photo, and determines the image regions corresponding to the faces of the object a, the object B, and the object C in the group photo, the first face image region may be determined according to the face distortion degree of the face image in each image region, for example, the image region with a higher face distortion degree may be determined as the first image region, so as to perform targeted processing subsequently.

Optionally, in this embodiment of the application, the human face image having human face distortion may include deformation of five sense organs of the human face, such as squinting, eye squinting, mouth skewness, and the like, and may include deformation of a face shape of the human face, such as facial skew, facial distortion, and the like.

Optionally, in this embodiment of the present application, the image processing apparatus may use a face key point identification algorithm to identify a face key point in the face image, and determine whether face distortion exists according to information such as a position and a pixel value of the identified face key point.

Illustratively, in combination with the foregoing embodiment, with the first face image region being an image region corresponding to the face of the object a in the first face image, the image processing apparatus may perform a face keypoint recognition algorithm on the face image in the image region corresponding to the face of the user a in the first face image region, detect coordinates of face keypoints in the face image, and determine a distance between keypoints corresponding to upper and lower eyelids of left and right eyes based on the coordinates of face keypoints, so as to determine whether there is a squinting situation on the face of the user a.

For convenience of description, in the embodiment of the present application, a face image in an image area corresponding to a face of a user a is recorded as a face image of the user a.

Specifically, the image processing apparatus may determine whether the face of the user a has squinting by comparing a size relationship between a distance between key points corresponding to the upper eyelid and the lower eyelid and the first distance threshold. Illustratively, the first threshold may be 0.3cm, 0.5cm, or 0.7cm, and the like, which is not limited in this embodiment of the application.

For example, when the distance between the key points corresponding to the upper eyelid and the lower eyelid in the face image is less than 0.5cm, it can be considered that the face of the user a has narrow eyes, that is, the face image has face distortion.

Optionally, in this embodiment of the application, in a case that it is detected that a face image in a first face image region has face distortion, the image processing apparatus may acquire a second face image, and perform image fusion processing on the first face image and the second face image to generate a target face image, where the target face image has no image region with face distortion.

Optionally, in this embodiment of the application, the second face image is an image obtained by shooting a target object and pre-stored locally in the electronic device or in a server, and the second face image is an image without face distortion and with high image quality. Therefore, the face correction can be carried out on the face image with poor quality of the target object based on the historical face image with good quality of the target object, and an image with good quality is generated.

Illustratively, the absence of face distortion means that five sense organs and facial shapes are not deformed or the deformation degree is lower than a preset degree, and the higher image quality means that the contour is clear and the definition is higher.

In the image processing method provided by the embodiment of the application, the image processing device determines at least one face image area in a first face image, and generates a target face image based on the first face image and a second face image under the condition that the face image in the first face image area has face distortion; the face image in the first face image area and the second face image area are face images including the same target object, and the second face image does not have face distortion. According to the method, the image processing device can generate the target face image based on the second face image and the first face image under the condition that the face distortion exists in the face area in the current first face image, and because the images in the face areas in the second face image and the first face image are the face images of the same object and the second face image does not have the face distortion, the target face image without the face distortion is generated by utilizing the historical face image with better quality of the target object and the face image with the face distortion existing in the target object at present, so that the face image with the face distortion existing in the face image can be accurately corrected, the face expression is greatly improved, and the image with higher quality is obtained.

Optionally, in this embodiment of the application, the generating of the target face image based on the first face image and the second face image in step 202 may include the following steps 202a1 and 202a2:

step 202a1: and performing face key point matching on the face image in the first face image region and the second face image, and determining a target region corresponding to a distortion region in the first face image region in the second face image.

Wherein the distortion region is an image region in which a human face is distorted in the first human face image region.

Step 202a2: and replacing the image in the distortion area with the image in the target area to generate a target face image and generate a target face image.

For example, the image processing apparatus may determine an image region in which a face distortion exists in the first face image region, that is, a distortion region, and then perform face keypoint matching on the face image in the first face image region and the second face image through a feature matching algorithm to determine an image region in the second face image corresponding to the distortion region.

Illustratively, the image processing apparatus acquires an image in a target region of the second face image, and replaces an image in a distorted region in the first face image with an image in the target region, thereby obtaining a high-quality face image in which the face distortion of the first face image is improved, i.e., a target face image.

For example, the image processing apparatus may use the pixel values of the respective pixels in the target region of the second face image as the pixel values of the respective pixels in the distorted region in the first face image region of the first face image to replace the image of the distorted region in the first face image region with the undistorted image of the target region to obtain the target face image.

For example, taking a first face image as an example of a single person photo captured for an object a, where the first face image region is an image region corresponding to a face of the object a in the first face image, and assuming that a right eye of the face in the first image region is in a closed-eye state, the image processing device may determine an image region corresponding to an image region where the right eye in the first image region is located in a historical face image of the object a, where the corresponding image region is an image region where the right eye in the second face image is located, and then replace an image in the image region where the right eye in the first image region is located with an image region where the right eye in the second face image is located, that is, replace an image in the right eye in the closed-eye state in the first image region with an image in the image region where the right eye in the second face image is located, so as to obtain an improved image without distortion, that is a target face image.

It should be noted that the historical face image is the second face image.

In some embodiments of the present application, the image processing apparatus may acquire an image of the target region, and then fuse the image of the target region to a distortion region in the first face image to obtain a fused high-quality face image, so as to improve face distortion of the first face image.

For example, the image processing apparatus may calculate an average value of gray levels of each pixel point of the target region of the second face image and a corresponding pixel point of the distorted region of the first face image, and then use the average value of gray levels as a gray level value of the fused pixel point, so as to fuse the image in the target region to the first face image region, so as to obtain a fused high-quality image, that is, the target face image.

In this way, the image processing apparatus may determine corresponding target regions in the first face image and the second face image, and replace or fuse image regions in which distortion occurs in the first face image region to obtain a target face image after face correction is performed on the face image in the first face image region, that is, the image processing method provided in the embodiment of the present application can perform correction processing on an image in which face distortion exists in the target object by using a history high-quality image of the target object, and can retain original face features, such as a nose, a mouth, and the like, in other image regions of the face image that do not need correction processing while correcting an image in a distortion region in the face image, thereby greatly improving accuracy and effectiveness of image processing.

Optionally, in this embodiment of the present application, before generating the target face image based on the first face image and the second face image in step 202, the image processing method provided in this embodiment of the present application further includes the following steps A1 and A2:

step A1: an image processing apparatus acquires a first data set.

Wherein the first data set comprises M third face images, and one third face image is a face image comprising an object; m is a positive integer.

Step A2: and the image processing device performs face key point matching on the face image in the first face image area and the M third face images, and determines a second face image matched with the face image in the first face image area in the M third face images.

The first data set may be a locally stored data set or a data set stored in a server.

Illustratively, the M third face images may be face images acquired for M objects.

Illustratively, the M third face images are face images without face distortion (or image distortion).

Illustratively, one third face image may include only the image of the face part of the object, so as to reduce the memory space occupied by the stored image and the operation amount of image processing.

For example, the image processing apparatus may perform face key point matching on the face image in the first face image region with each third face image through a face key point matching algorithm, and determine a second face image from the third face images, where a matching degree with the face image in the first face image region is greater than a matching degree threshold, so as to perform image correction on the face image in the first face image region subsequently using the second face image.

For example, taking the M third face images including the face image a collected for the object a, the face image B collected for the object B, and the face image C collected for the object C as an example, the image processing apparatus may perform face key point matching on the face image in the first face image region with the face image a, the face image B, and the face image C, respectively, and determine the face image a as the target face image if the degree of matching of the face image a, the face image B, and the face image C with the face image in the first face image region is greater than 90%, and determine the face image a as the target face image if the degree of matching of the face image a, the face image B, and the face image C with the face image in the first face image region is 96%,30%, and 25%, respectively.

It should be noted that the more similar the faces in the two face images are, the higher the matching degree of the key points of the two face images is, so that the face image with the higher matching degree with the face image of the target object in the first face image region can be regarded as another face image shot for the target object.

It should be noted that the target object in the embodiments of the present application, for example, object a, refers to a human.

Therefore, the image processing device can determine the face image matched with the face image in the first face image area from the plurality of face images with good quality, so that the historical high-quality same-person photo matched with the face image can be determined, and the face image can be accurately corrected in the follow-up process.

Further optionally, in this embodiment, before the step A1, the image processing method provided in this embodiment further includes the following steps B1 to B3:

step B1: the image processing apparatus acquires N second data sets.

Wherein each second data set comprises a plurality of face images acquired for one object.

And step B2: and the image processing device respectively determines fourth face images of which the face key points in the N second data sets meet preset conditions to obtain N fourth face images.

And step B3: the image processing device generates a first data set based on the N fourth face images.

Illustratively, the N second data sets are data sets stored locally by the electronic device or data sets stored in a server.

Illustratively, taking the example that the N second data sets include three data sets, the N second data sets may include a data set 1, a data set 2, and a data set 3, where the data set 1 includes a plurality of face images taken with respect to a subject a, the data set 2 includes a plurality of face images taken with respect to a subject B, and the data set 3 includes a plurality of face images taken with respect to a subject C. Therefore, the image processing device can obtain the data set of the specially-assigned person so as to process the image of the specially-assigned person aiming at the data set of the specially-assigned person in the follow-up process and improve the accuracy of image processing.

Illustratively, the second data set is obtained by classifying the third data set.

Illustratively, the image processing apparatus may acquire a large number of images including faces from a user album or download a large number of images including faces from a network, detect positions of faces in the images using a face detection algorithm, extract face images in a face region in the images, construct a third data set according to the extracted face images, perform face recognition on the face images using a face recognition algorithm, and classify the face images in the third data set according to face features in the face images to obtain the N second data sets.

Illustratively, the preset condition may include at least one of: the number of the face key points is larger than a first threshold, the number of the face key points of the target characteristic part is larger than a preset number threshold, and the distance between the face key points of the target characteristic part is within a preset distance range. Illustratively, the above-mentioned feature may be an eye, a nose, a mouth, a facial contour, and the like.

For example, taking the key points of the human face as the key points of the upper eyelid and the lower eyelid, the condition that the key points of the human face meet the preset conditions may be: the distance between the key points of the upper eyelid and the lower eyelid is in a preset distance range, which may be, for example, a distance between 0.5cm and 1.5 cm.

The image processing apparatus may perform processing such as correction processing or feature recognition on an image based on position information of key points by detecting key points of a face and determining key points of each feature of the face.

It can be understood that the fourth face image whose face key points satisfy the preset condition is a face image whose feature parts (for example, five sense organs) have position standards and whose feature parts have no deformation, that is, a face image whose face has no deformation and whose image quality is better.

For example, in a case where a fourth face image satisfying a preset condition in each data set is determined, the image processing apparatus may construct the first data set based on the fourth face image satisfying the preset condition to obtain a data set including a better quality face image of each object, so as to facilitate performing face correction processing on the first face image to be processed subsequently through the better quality face image.

Optionally, in this embodiment of the application, before generating the target face image based on the first face image and the second face image in step 202, the image processing method provided in this embodiment of the application further includes the following steps C1 and C2:

step C1: the image processing apparatus acquires N second data sets.

Wherein each second data set comprises a plurality of face images, each face image being a face image comprising an object. That is, each second data set comprises a plurality of face images acquired for one object.

And step C2: and the image processing device trains the N neural network models respectively based on the N second data sets to obtain the trained N neural network models.

Wherein one neural network model corresponds to one second data set.

It should be noted that, the description of the third data set can be referred to above, and is not repeated herein.

Fig. 2 is a schematic diagram of a data set provided in an embodiment of the present application. The third data set and the second data set of the embodiment of the present application are explained below with reference to fig. 2. As shown in fig. 2, the third data set includes

photos

1, 2, 3, etc., and the image processing apparatus first executes a face detection algorithm on the

photos

1, 2, 3, etc., to determine face regions in the

photos

1, 2, 3, then executes a face key point recognition algorithm on the face regions in the

photos

1, 2, 3, and classifies the photos according to different faces to obtain the

photos

1, 2, 3 of the user a, 4, 5, and other photos of the user B, and finally superimposes blur and noise on each photo of the user a and the user B to obtain a low-quality photo corresponding to each photo, thereby constructing a low-quality photo album 11 of the user a and a low-quality photo album 12 of the user B.

For example, the N neural network models may be face-enhanced neural network models.

For example, the image processing apparatus may perform image degradation processing on a plurality of face images in each second data set to obtain a low-quality face image corresponding to each face image, construct an image pair of an original face image before degradation and the low-quality face image to obtain a plurality of image pairs, and then train the neural network model by using the plurality of image pairs as input of the neural network model to obtain a trained face enhanced neural network model.

Illustratively, when neural network model training is performed, a network architecture adopts a common U-NET, input and output data pairs adopt an online generation mode, a training Loss function (Loss) adopts a Loss combination mode to obtain a better effect, and the training Loss function (Loss) comprises brightness Loss (brightness Loss), gradient Loss (gradient Loss), color Loss (color Loss), perception Loss (perceptual Loss) and the like, and an ADAM optimizer is adopted for iterative optimization.

Illustratively, an online generation mode of input and output data in the neural network model training process is as follows, for each second data set, an original image is taken out each time, a Gaussian fuzzy kernel with a certain size is randomly applied to the original image to obtain a fuzzy degraded image, random Gaussian noise and Poisson noise are added to obtain a noise degraded image corresponding to the original image, the noise degraded image is used as the input of the neural network, a low-quality photo set is generated through a large amount of random degradation, the original non-degraded image is used as a truth value (GT) of the network, loss is calculated by using the network output and the GT, a face enhancement model can be obtained through a large amount of iterations, and finally, a specially-assigned face AI enhancement algorithm model, namely the trained neural network model, is obtained.

Fig. 3 is a schematic diagram of a neural network model training process provided in an embodiment of the present application, and the neural network model training process is described below by taking a second data set including a data set 1 and a data set 2 as an example, as shown in fig. 2, the data set 1 includes a picture 1 of a user a, a picture 2 of the user a, and a picture 3 of the user a, the image processing apparatus superimposes blur, gaussian, and poisson noise on the picture 1 of the user a, the picture 2 of the user a, and the picture 3 of the user a, respectively, to obtain a low-quality picture set 11 of the user a, and superimposes gaussian blur, gaussian, and poisson noise on the picture 1 of the user B and other pictures to obtain a low-quality picture set 12 of the user B, and uses the low-quality picture set of the user a and the low-quality picture set of the user B as inputs of an AI network a and an AI network B, outputs a prediction map corresponding to the pictures in the low-quality picture set, uses an original undegraded image as a true value of the network, then calculates loss through a loss function, adjusts loss based on a loss function and a loss function, and a large amount of the AI network training process after the AI network training, and the neural network training, and the network training process N.

For ease of description, the photograph of user a may be recorded as an a-person photograph and the photograph of user B may be recorded as a B-person photograph.

It can be understood that, because the trained neural network model is the neural network model for image enhancement for the face image of the special person, the AI face enhancement algorithm model of the special person can process the face image with lower quality of the special person, and can perform image enhancement on the face image of the special person more specifically, so that the image can be accurately optimized, and the face image with better image quality can be obtained.

It should be noted that the above-mentioned face image with lower quality may be a blurred face image.

Further optionally, in this embodiment of the present application, after the target face image is generated in step 202, the image processing method provided in this embodiment of the present application further includes the following step C1:

step C1: and the image processing device determines a target neural network model and performs image enhancement processing on the target face image through the target neural network model to obtain a processed target face image.

The target neural network model is a neural network model corresponding to a target data set in the trained N neural network models, and the target data set is a data set of the N second data sets, wherein the data set comprises a face image acquired aiming at a target object.

For example, the image processing apparatus may determine a special person neural network model corresponding to the target object, and perform image enhancement processing on the target face image through the special person neural network model, so as to obtain a face image with better quality after denoising and deblurring the target face image.

Exemplarily, with the above embodiment, taking the target data set as the data set 1 of the user a as an example, the image processing apparatus may obtain an AI network a corresponding to the user a, and perform image enhancement processing on the target face image through the AI network a, so as to obtain a portrait photo with better facial expression of the user a and better quality of denoising and deblurring.

The user a is the target object.

Fig. 4 is a flowchart of an image processing method according to an embodiment of the present application, and as shown in fig. 4, an image processing apparatus obtains multiple photos in an album, then performs face detection on the multiple photos, scratches out a face region, constructs a data set 1, that is, a third data set, and then performs face recognition on the data set 1 to obtain a data set of a user a, that is, a second data set, which is denoted as A2 data set, and a data set of a user B, that is, a second data set, which is denoted as a B2 data set. Then, the image processing device can perform noise and fuzzy processing on the images in the A2 data set and the B2 data set to obtain corresponding degraded images, and then respectively train the network A and the network B based on the degraded images and the original images before degradation to obtain the trained network A and the trained network B; and the image processing device may perform face key point detection on the images in the A2 data set and the B2 data set, calculate a maximum distance between two points of the upper and lower eyelids in the image, determine an image with the best eye effect, that is, an image with a more natural eye expression, from the A2 data set and the B2 data set, respectively, based on the maximum distance between the two points of the upper and lower eyelids in the image, generate A3 data and B3 data, and construct a data set 3, that is, a first data set, based on the A3 data and the B3 data, where the image in the data set 3 is an image without distortion in an eye region, that is, an image with a more natural eye expression.

According to the image processing method provided by the embodiment of the application, the problem of poor facial expression in group photo is avoided by using the photos with natural historical expressions, and the photos with high historical quality of the same person are used as training data, so that the details and noise expression of the face are improved, the original skin detail characteristics can be effectively kept, and the negative effects that the details are false and the characteristic nevus are removed as noise are not easy to introduce are achieved.

In the image processing method provided by the embodiment of the application, the execution subject can be an image processing device. In the embodiment of the present application, an image processing apparatus is taken as an example to execute an image processing method, and the image processing apparatus provided in the embodiment of the present application is described.

Fig. 5 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application, and as shown in fig. 5, the image processing apparatus 500 may include a determining module 501 and a generating module 502, where the determining module 501 is configured to determine at least one facial image area in a first facial image; the generating module 502 is configured to generate a target face image based on the first face image and the second face image when a face image in the first face image region has face distortion; the face image in the first face image region and the second face image region are face images including the same target object, and the second face image does not have face distortion.

Optionally, in an embodiment of the present application, the generating module is specifically configured to perform face keypoint matching on a face image in the first face image region and a second face image, and determine a target region in the second face image, where the target region corresponds to a distortion region in the first face image region, where the distortion region is an image region in which a face is distorted in the first face image region;

the generating module is specifically configured to replace the image in the distorted region with the image in the target region, and generate a target face image.

Optionally, in an embodiment of the present application, the apparatus further includes: an acquisition module and a determination module, wherein:

the acquisition module is used for acquiring a first data set, wherein the first data set comprises M third face images, and one third face image is a face image comprising an object; m is a positive integer;

the determining module is configured to perform face key point matching on the face image in the first face image region and the M third face images, and determine a second face image, which is matched with the face image in the first face image region, in the M third face images.

Optionally, in an embodiment of the present application, the apparatus further includes: a training module;

the acquiring module is further configured to acquire N second data sets, where each second data set includes a plurality of face images, and each face image is a face image including an object;

the training module is configured to train N neural network models respectively based on the N second data sets acquired by the acquisition module to obtain N trained neural network models, where one neural network model corresponds to one second data set.

Optionally, in an embodiment of the present application, the apparatus further includes: a processing module;

the determining module is also used for determining a target neural network model;

the processing module is used for carrying out image enhancement processing on the target face image through the target neural network model determined by the determining module to obtain the processed target face image;

the target neural network model is a neural network model corresponding to a target data set in the trained N neural network models, and the target data set is a data set of the N second data sets, wherein the data set comprises a face image acquired aiming at the target object.

In the image processing apparatus provided by the embodiment of the application, the image processing apparatus determines at least one face image area in a first face image, and generates a target face image based on the first face image and a second face image when face distortion exists in the face image in the first face image area; the face image in the first face image area and the second face image area are face images including the same target object, and the second face image does not have face distortion. By the method, the image processing device can generate the target face image based on the second face image and the first face image under the condition that the face area in the current first face image has face distortion, and because the images in the face areas in the second face image and the first face image are the face images of the same object and the second face image does not have face distortion, the target face image without face distortion is generated by utilizing the historical face image with better quality of the target object and the face image with face distortion of the target object, so that the face image with face distortion in the face image can be accurately corrected, the face expression is greatly improved, and the image with higher quality is obtained.

The image processing apparatus in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be a device other than a terminal. The electronic Device may be, for example, a Mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic Device, a Mobile Internet Device (MID), an Augmented Reality (AR)/Virtual Reality (VR) Device, a robot, a wearable Device, an ultra-Mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and may also be a server, a Network Attached Storage (Network Attached Storage, NAS), a personal computer (NAS), a Television (TV), a teller machine, a self-service machine, and the like, and the embodiments of the present application are not limited in particular.

The image processing apparatus in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, which is not specifically limited in the embodiment of the present application.

The image processing apparatus provided in the embodiment of the present application can implement each process implemented by the method embodiments of fig. 1 to fig. 4, and is not described herein again to avoid repetition.

Optionally, as shown in fig. 6, an electronic device 600 is further provided in an embodiment of the present application, and includes a processor 601 and a memory 602, where a program or an instruction that can be executed on the processor 601 is stored in the memory 602, and when the program or the instruction is executed by the processor 601, the steps of the embodiment of the image processing method are implemented, and the same technical effects can be achieved, and are not described again here to avoid repetition.

It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 7 is a schematic diagram of a hardware structure of an electronic device implementing the embodiment of the present application.

The electronic device 100 includes, but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

Those skilled in the art will appreciate that the electronic device 100 may further comprise a power source (e.g., a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in fig. 7 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.

Wherein the processor 110 is configured to determine at least one facial image region in the first facial image; the processor 110 is configured to generate a target face image based on the first face image and the second face image when a face distortion exists in a face image in the first face image region; the face image in the first face image region and the second face image region are face images including the same target object, and the second face image does not have face distortion.

Optionally, in this embodiment of the application, the processor 110 is specifically configured to perform face keypoint matching on a face image in the first face image region and a second face image, and determine a target region in the second face image, where the target region corresponds to a distortion region in the first face image region, where the distortion region is an image region in which a face is distorted in the first face image region;

the processor 110 is specifically configured to replace the image in the distorted area with the image in the target area, so as to generate a target face image.

Optionally, in this embodiment of the present application, the processor 110 is configured to obtain a first data set, where the first data set includes M third facial images, and one third facial image is a facial image that includes one object; m is a positive integer;

the processor 110 is configured to perform face key point matching on the face image in the first face image region and the M third face images, and determine a second face image in the M third face images, where the second face image matches the face image in the first face image region.

Optionally, in this embodiment of the present application, the processor 110 is further configured to acquire N second data sets, where each second data set includes a plurality of face images, and each face image is a face image including an object;

the processor 110 is configured to train N neural network models based on the N second data sets, respectively, to obtain N trained neural network models, where one neural network model corresponds to one second data set.

Optionally, in this embodiment of the present application, the processor 110 is further configured to determine a target neural network model;

the processor 110 is configured to perform image enhancement processing on the target face image through the target neural network model determined by the processor 110, so as to obtain a processed target face image;

the target neural network model is a neural network model corresponding to a target data set in the trained N neural network models, and the target data set is a data set including face images acquired aiming at the target object in the N second data sets.

According to the electronic equipment provided by the embodiment of the application, the electronic equipment determines at least one face image area in a first face image, and generates a target face image based on the first face image and a second face image under the condition that the face image in the first face image area has face distortion; the face image in the first face image area and the second face image area are face images including the same target object, and the second face image does not have face distortion. By the method, the electronic equipment can generate the target face image based on the second face image and the first face image under the condition that the face distortion exists in the face area in the current first face image, and because the images in the face areas in the second face image and the first face image are the face images of the same object and the second face image does not have the face distortion, the target face image without the face distortion is generated by utilizing the historical face image with better quality of the target object and the face image with the face distortion existing at the current target object, so that the face image with the face distortion existing in the face image can be accurately corrected, the face expression is greatly improved, and the image with higher quality is obtained.

It should be understood that, in the embodiment of the present application, the input Unit 104 may include a Graphics Processing Unit (GPU) 1041 and a microphone 1042, and the Graphics Processing Unit 1041 processes image data of a still picture or a video obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also referred to as a touch screen. The touch panel 1071 may include two parts of a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein.

The memory 109 may be used to store software programs as well as various data. The memory 109 may mainly include a first storage area storing a program or an instruction and a second storage area storing data, wherein the first storage area may store an operating system, an application program or an instruction (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, memory 109 may comprise volatile memory or non-volatile memory, or memory 109 may comprise both volatile and non-volatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. The volatile Memory may be a Random Access Memory (RAM), a Static Random Access Memory (Static RAM, SRAM), a Dynamic Random Access Memory (Dynamic RAM, DRAM), a Synchronous Dynamic Random Access Memory (Synchronous DRAM, SDRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, ddr SDRAM), an Enhanced Synchronous SDRAM (ESDRAM), a Synchronous Link DRAM (SLDRAM), and a Direct bus RAM (DRRAM). The memory 109 in the embodiments of the subject application includes, but is not limited to, these and any other suitable types of memory.

Processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor, which mainly handles operations related to the operating system, user interface, application programs, etc., and a modem processor, which mainly handles wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 110.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the embodiment of the image processing method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory ROM, a random access memory RAM, a magnetic or optical disk, and the like.

The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the embodiment of the image processing method, and can achieve the same technical effect, and is not described here again to avoid repetition.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.

Embodiments of the present application provide a computer program product, where the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the processes of the foregoing image processing method embodiments, and can achieve the same technical effects, and in order to avoid repetition, details are not repeated here.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a component of' 8230; \8230;" does not exclude the presence of another like element in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the present embodiments are not limited to those precise embodiments, which are intended to be illustrative rather than restrictive, and that various changes and modifications may be effected therein by one skilled in the art without departing from the scope of the appended claims.

Claims

1. An image processing method, characterized in that the method comprises:

determining at least one face image region in a first face image;

under the condition that the face image in the first face image area has face distortion, generating a target face image based on the first face image and the second face image;

the face image in the first face image region and the second face image region are face images including the same target object, and the second face image does not have face distortion.

2. The method of claim 1, wherein generating the target facial image based on the first facial image and the second facial image comprises:

performing face key point matching on a face image in the first face image region and a second face image, and determining a target region corresponding to a distortion region in the first face image region in the second face image, wherein the distortion region is an image region with face distortion in the first face image region;

and replacing the image in the distortion area with the image in the target area to generate a target face image.

3. The method of claim 1, wherein before generating the target facial image based on the first facial image and the second facial image, further comprising:

acquiring a first data set, wherein the first data set comprises M third face images, and one third face image is a face image comprising an object; m is a positive integer;

and performing face key point matching on the face image in the first face image area and the M third face images, and determining a second face image matched with the face image in the first face image area in the M third face images.

4. The method of claim 1, wherein before generating the target facial image based on the first facial image and the second facial image, further comprising:

acquiring N second data sets, wherein each second data set comprises a plurality of face images, and each face image is a face image comprising an object;

and training the N neural network models respectively based on the N second data sets to obtain the trained N neural network models, wherein one neural network model corresponds to one second data set.

5. The method of claim 4, wherein after generating the target face image, the method further comprises:

determining a target neural network model, and performing image enhancement processing on the target face image through the target neural network model to obtain the processed target face image;

6. An image processing apparatus, characterized in that the apparatus comprises: a determination module and a generation module, wherein:

the determining module is used for determining at least one face image area in the first face image;

the generating module is used for generating a target face image based on the first face image and the second face image under the condition that the face image in the first face image area has face distortion;

the face image in the first face image area and the second face image area comprise face images collected by the same target object, and face distortion does not exist in the second face image.

7. The apparatus of claim 6,

the generating module is specifically configured to perform face key point matching on a face image in the first face image region and a second face image, and determine a target region in the second face image corresponding to a distortion region in the first face image region, where the distortion region is an image region in which face distortion exists in the first face image region;

8. The apparatus of claim 6, further comprising: an acquisition module and a determination module, wherein:

the acquisition module is used for acquiring a first data set, wherein the first data set comprises M third face images, and one third face image is a face image acquired aiming at one object; m is a positive integer;

9. The apparatus of claim 6, further comprising: a training module;

10. The apparatus of claim 9, further comprising: a processing module;

the target neural network model is a neural network model corresponding to a target data set in the trained N neural network models, and the target data set is a data set including face images of the target object in the N second data sets.