WO2020252910A1

WO2020252910A1 - Image distortion correction method, apparatus, electronic device and readable storage medium

Info

Publication number: WO2020252910A1
Application number: PCT/CN2019/102870
Authority: WO
Inventors: 叶唐陟; 吴棨贤
Original assignee: 厦门美图之家科技有限公司
Priority date: 2019-06-17
Filing date: 2019-08-27
Publication date: 2020-12-24
Also published as: CN110232667B; CN110232667A

Abstract

An image distortion correction method, an apparatus, an electronic device and a readable storage medium, wherein the method comprises the steps of performing face recognition on an image to be corrected to obtain face frame information and face key points corresponding to at least two faces in the image to be corrected (S110), then determining a face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculating a relative distance coefficient between the face to be corrected and a camera lens (S120), and then performing distortion correction on the face to be corrected according to the relative distance coefficient to obtain a target image after the distortion correction (S130). The method can automatically recognize a face to be corrected and perform distortion correction in real-time when multiple people take a selfie together, thereby optimizing the shooting effect in a multi-person selfie scene.

Description

Image distortion correction method, device, electronic equipment and readable storage medium

Cross references to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on June 17, 2019, with the application number 2019105217671, titled "Image Distortion Correction Method, Device, Electronic Equipment, and Readable Storage Medium", the entire content of which is approved The reference is incorporated in this application.

Technical field

This application relates to the technical field of graphics and image processing, and in particular to an image distortion correction method, device, electronic equipment, and readable storage medium.

Background technique

At present, using the front camera of an electronic device (such as a smart phone, a tablet computer, etc.) to take a self-portrait photo with multiple people is an important use scene in taking pictures. However, due to the limitation of the length of the human arm, the physical distance of the photographer from the lens usually cannot exceed the length of the arm, while the other people in the group photo do not have the physical limitation of hand-held electronic devices, and can freely choose a suitable location for shooting. At this time, the photographer in the group photo is usually the person closest to the lens, which will cause the face of the photographer holding the electronic device to be distorted when multiple people take a group photo. This distortion will be more obvious when a group photo is taken. As a result, in a scene where multiple people take a selfie together, the face of the photographer will look abnormal compared to other people, which will affect the shooting effect.

Summary of the invention

In view of this, one of the objectives of the embodiments of the present application is to provide an image distortion correction method, device, electronic device, and readable storage medium, which can automatically recognize the face to be corrected for distortion correction in real time when multiple people take a selfie together. In order to optimize the shooting effect in the multi-person selfie photo scene.

The present application provides an electronic device, which may include one or more storage media and one or more processors in communication with the storage media. One or more storage media stores machine executable instructions executable by the processor. When the electronic device is running, the processor executes the machine executable instructions to execute the image distortion correction method.

The present application also provides an image distortion correction method, which is applied to electronic equipment, and the method includes:

Performing face recognition on the image to be corrected to obtain face frame information and face key points corresponding to at least two faces in the image to be corrected;

Determine the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculate the relative distance coefficient between the face to be corrected and the camera lens;

Distortion correction is performed on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction.

Optionally, the step of performing face recognition on the image to be corrected to obtain face frame information and face key points corresponding to at least two faces in the image to be corrected includes:

After detecting the camera start command, turn on the camera and enter the shooting preview interface;

For each frame of the image to be corrected in the shooting preview interface, face recognition is performed on the frame of the image to be corrected through the face recognition model obtained in advance to obtain the face frame information corresponding to each face in the frame of the image to be corrected And key points of the face;

Wherein, the face recognition model uses multiple training samples and the labeled data of each training sample to be obtained based on deep learning neural network training, where the labeled data of each training sample includes the face frame corresponding to each face in the training sample Key points of information and faces.

Optionally, before the step of determining the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculating the relative distance coefficient between the face to be corrected and the camera lens , The method further includes:

For each face, crop out the corresponding face image according to the face frame information corresponding to the face;

According to the key points of the face corresponding to the face, the face image is rotated to the set position using the affine matrix.

Optionally, the method further includes:

For each face image after rotation, a pre-trained age estimation model is used to recognize the face image to obtain the face age in the face image;

Determine whether the age of the face in the face image is greater than the set age;

If the age of the face in the face image is less than the set age, the size of the face frame of the face image is corrected according to the age of the face in the face image.

Optionally, the electronic device pre-stores the median of the face circumference corresponding to the age of each face, and the size of the face frame of the face image is performed according to the face age in the face image. The corrective steps include:

Acquiring the first median of the face circumference corresponding to the age of the face in the face image and the second median of the face circumference corresponding to the set age;

Calculating a face frame correction coefficient according to the first median and the second median;

The size of the face frame of the face image is corrected according to the face frame correction coefficient.

Optionally, the step of determining the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculating the relative distance coefficient between the face to be corrected and the camera lens, include:

According to the face frame information corresponding to each face, determine the face with the largest face frame area as the face to be corrected;

Calculate the relative distance coefficient between the face to be corrected and the camera lens.

Optionally, the step of calculating the relative distance coefficient between the face to be corrected and the camera lens includes:

Calculating the average area of the face frame corresponding to all recognized faces according to the face frame information corresponding to each face;

Calculating the sum of the square value of the difference between the area of the face frame corresponding to each face and the average area;

The relative distance coefficient between the face to be corrected and the camera lens is calculated according to the sum of the squared values of the difference. The specific calculation formula is:

Where, d is the relative distance coefficient between the face to be corrected and the camera lens, N is the number of faces, x ⁱ is the area of the ith face frame, and r is the average area of the face frame corresponding to all faces.

Acquiring the median area and the maximum number area corresponding to the face frame of the face other than the face to be corrected;

Calculating a first ratio of the median area to the face frame area of the face to be corrected and a second ratio of the maximum number area to the face frame area of the face to be corrected;

The relative distance coefficient between the face to be corrected and the camera lens is calculated according to the weight coefficients corresponding to the preset first ratio and second ratio, the first ratio and the second ratio, and the specific calculation formula is:

Where d is the relative distance coefficient between the face to be corrected and the camera lens, a _max is the maximum number area corresponding to the face frame of the face other than the face to be corrected, and a _mid is the face to be corrected The median area corresponding to the face frame of the face other than the face, K is a constant between 0 and 1, and y is the face frame area of the face to be repaired.

Optionally, the electronic device pre-stores a plurality of distortion correction parameters corresponding to preset distance coefficients, and the distortion correction is performed on the face to be corrected according to the relative distance coefficients to obtain the target image after the distortion correction. The steps include:

Acquiring a preset distance coefficient range in which the relative distance coefficient is located, the preset distance coefficient range including a first endpoint value and a second endpoint value, the first endpoint value is less than the second endpoint value;

Calculate the first difference between the relative distance coefficient and the first endpoint value, the second difference between the second endpoint value and the first endpoint value, and the distortion correction corresponding to the second endpoint value The third difference between the parameter and the distortion correction parameter corresponding to the first endpoint value;

Calculating corresponding target distortion correction parameters according to the first endpoint value, the first difference value, the second difference value, and the third difference value;

Performing distortion correction on the face to be corrected according to the target distortion correction parameter to obtain a target image after the distortion correction;

Wherein, the target distortion correction parameter is obtained by the following calculation formula:

Where z is the target distortion correction parameter, d is the relative distance coefficient between the face to be corrected and the camera lens, d ₁ is the first endpoint value, d ₂ is the second endpoint value, and c ₁ is the The distortion correction parameter corresponding to the first endpoint value, and c ₂ is the distortion correction parameter corresponding to the first endpoint value.

Optionally, the step of performing distortion correction on the face to be corrected according to the target distortion correction parameter to obtain a target image after the distortion correction includes:

Establishing a face grid of the face to be corrected, and determining each constraint point in the face grid;

Calculating the constraint deformation variables of each constraint point in the face grid according to the target distortion correction parameter;

Adjust the coordinates of each constraint point according to the calculated constraint deformation variables of each constraint point to obtain an adjusted face grid;

The face to be corrected is mapped to the adjusted face grid to obtain a target image after distortion correction.

Optionally, the method further includes:

Display each target image after distortion correction in the shooting preview interface in real time, and when a shooting instruction is detected, use the target image currently displayed in the shooting preview interface as a shooting image and store it in the electronic device in.

Optionally, the step of determining the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face includes:

According to the face frame information corresponding to each face, it is determined that the face with the largest face frame area is the face to be corrected.

The relative distance coefficient between the face to be corrected and the camera is calculated according to the area of the face frame of each face.

Optionally, before the determining that the face with the largest face frame area is the face to be corrected, the method further includes:

Identifying the age of the face corresponding to each face image in the image to be corrected;

The size of the face frame of each face image is corrected according to the face age corresponding to each face image.

The present application also provides an image distortion correction device applied to electronic equipment, and the device includes:

A recognition module configured to perform face recognition on the image to be corrected, and obtain face frame information and face key points corresponding to at least two faces in the image to be corrected;

A calculation module, configured to determine the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculate the relative distance coefficient between the face to be corrected and the camera lens;

The distortion correction module is configured to perform distortion correction on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction.

The present application also provides a readable storage medium having machine executable instructions stored on the readable storage medium, and the computer program can execute the steps of the above-mentioned image distortion correction method when the computer program is run by the processor.

Based on the foregoing, the embodiment of the present application obtains face frame information and face key points corresponding to at least two faces in the image to be corrected by performing face recognition on the image to be corrected, and then according to the face corresponding to each recognized face The frame information determines the face to be corrected in the image to be corrected, and calculates the relative distance coefficient between the face to be corrected and the camera lens, and then performs distortion correction on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction. In this way, it is possible to automatically recognize the face to be corrected for distortion correction in real-time when multiple people take a selfie together, thereby optimizing the shooting effect in a scene where multiple people take a selfie together.

Description of the drawings

In order to more clearly describe the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore do not It should be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative work.

FIG. 1 shows one of the schematic flowcharts of the image distortion correction method provided by the embodiment of the present application;

FIG. 2 shows a schematic diagram of a face frame corresponding to a recognized face provided by an embodiment of the present application;

FIG. 3 shows the second schematic flowchart of the image distortion correction method provided by the embodiment of the present application;

4 shows a schematic diagram of a photographing preview interface of an electronic device before image distortion correction provided by an embodiment of the present application;

FIG. 5 shows a schematic diagram of a photographing preview interface of an electronic device after image distortion correction provided by an embodiment of the present application;

FIG. 6 shows one of the schematic block diagrams of the functional modules of the image distortion correction device included in the electronic device provided by the embodiment of the present application;

FIG. 7 shows the second schematic block diagram of the functional modules of the image distortion correction apparatus included in the electronic device provided by the embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described clearly and completely in conjunction with the drawings in the embodiments of this application. It should be understood that this application is attached The drawings are only for the purpose of illustration and description, and are not used to limit the protection scope of this application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowchart used in this application shows operations implemented according to some embodiments of the embodiments of this application. It should be understood that the operations of the flowchart may be implemented out of order, and steps without logical context may be reversed in order or implemented at the same time. In addition, under the guidance of the content of this application, those skilled in the art can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present application.

The inventor of the present application has discovered through research that in a scene where multiple people are photographed together, the photographer will distort his face when he is close to the lens, the face area will be enlarged, and the other people in the group photo will be relative to the photographer. Usually the distance from the lens is farther, which will cause the distortion of the photographer's face area to be enlarged in the contrast. Current distortion correction methods mostly use general face-lifting or head-shrinking schemes to reduce distortion, but they cannot automatically perform distortion processing on the photographers in the group photo in real-time for multi-person selfie photo scenes.

To this end, based on the discovery of the above technical problems, the inventor proposes the following technical solutions to solve or improve the above problems. It should be noted that the defects in the above solutions in the prior art are the results of the inventors after practice and careful study. Therefore, the discovery process of the above problems and the following embodiments of the application address the above problems. The proposed solutions should all be contributions made by the inventor to the application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

Figure 1 shows a schematic flow chart of an image distortion correction method provided by an embodiment of the present application. It should be understood that in other embodiments, the sequence of some steps of the image distortion correction method of this embodiment may not be specifically implemented as shown in Figure 1 and below. The order of the examples is limited, for example, they can be exchanged according to actual needs, or some of the steps can also be omitted or deleted. The detailed steps of the image distortion correction method are introduced as follows.

Step S110: Perform face recognition on the image to be corrected to obtain face frame information and face key points corresponding to at least two human faces in the image to be corrected.

Step S120: Determine the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculate the relative distance coefficient between the face to be corrected and the camera lens.

Step S130, performing distortion correction on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction.

Based on the above steps, this embodiment can automatically recognize the face to be corrected in real time when multiple people take a selfie together to correct the distortion based on the calculated relative distance coefficient between the face to be corrected and the camera lens, thereby optimizing the multi-person selfie scene The shooting effect.

As a possible implementation manner, for step S110, after the camera start instruction is detected, the camera is turned on and the shooting preview interface is entered.

Among them, in different application scenarios, the manner of detecting the camera start instruction may be different. For example, the camera can be turned on when the camera control triggered by the photographer on the interactive interface is detected; for example, the camera can be turned on when the camera voice command sent by the photographer is received; for example, you can also When it is detected that the action of the photographer is consistent with the preset action of taking a picture, a camera start instruction is obtained.

When the camera start instruction is detected, the camera is turned on and enters the shooting preview interface. In the shooting preview interface, the image of the current shooting scene acquired by the camera can be displayed in real time.

In order to facilitate the photographer to see the face effect after distortion correction in real time during the shooting preview stage, the image to be corrected may be each frame of the image displayed in the shooting preview interface. On this basis, for each frame of the image to be corrected in the shooting preview interface, face recognition can be performed on the frame of the image to be corrected through the face recognition model obtained in advance to obtain each person in the frame of the image to be corrected The face frame information and key points of the face corresponding to the face.

Optionally, the face recognition model can be obtained by training based on a deep learning neural network (for example, YOLO neural network, Fast-RCNN neural network, MTCNN neural network, etc.) using multiple training samples and label data of each training sample. The labeled data of each training sample may include face frame information and face key points corresponding to each face in the training sample.

As an example, the face frame information corresponding to each face may include face ID information, coordinate information of the vertices of the face frame, width information and height information of the face frame, etc. The key points of the face may include various parts of the face. The feature points and the geometric relationship between each feature point.

For example, referring to FIG. 2 in conjunction, a schematic diagram of a face frame F corresponding to a recognized face is shown. In Figure 2, the face frame F covers the face area, and the face frame information may include the width W, height H, and vertex coordinates Q of the face frame shown in Figure 2, and the face frame F The area is the product of the width W and the height H of the face frame F. Among them, for faces of different sizes, the width W and height H of the face frame F are adaptively adjusted during the recognition process. In addition, the vertex coordinate Q of the face frame F can be selected according to actual needs. For example, the vertex coordinate Q shown in FIG. 2 is the vertex coordinate of the upper right corner of the face frame. In other examples, the vertex coordinate Q of the face frame can also be selected. Select the vertex coordinates of the upper left corner, the vertex coordinates of the lower left corner, or the vertex coordinates of the lower right corner, or any combination of the foregoing, and this embodiment does not impose any limitation on this.

In this way, the face recognition model obtained through neural network training based on deep learning can identify the face frame information and face key points corresponding to each face in the image to be corrected, so as to facilitate subsequent use of the face corresponding to each face The frame information and key points of the face are corrected for distortion.

The inventor also discovered in the process of research that usually in the video stream in the above-mentioned shooting preview interface, these faces may exist due to the deviation of the head of the subject, the camera shake, the relative position of the lens, etc. In the case of skew, the subsequent determination of the area of the face image will affect the distortion correction effect of the image.

Based on this, as a possible implementation manner, before step S120, for each face, a corresponding face image can be cropped according to the face frame information corresponding to the face. In detail, the cropping area of the face can be determined according to the coordinate information of the vertices of the face frame corresponding to the face, the width information and the height information of the face frame, and then the corresponding face image can be cropped according to the determined cropping area.

Then, according to the key points of the face corresponding to the face, the face image is rotated to the set position using the affine matrix. In detail, the difference between each face key template point in the standard face template stored in advance and the face key point corresponding to the face can be used to calculate the rotation parameter for rotating the face image using the affine matrix, Then, the face key points corresponding to the face image are rotated to the corresponding positions determined according to the rotation parameters to obtain the face image after the straightening. In this way, it can be avoided that these faces may be skewed due to the deflection of the head of the subject, the camera shake, or the offset of the relative position of the lens.

The inventor also discovered in the process of research that usually when there are children in a group photo, due to the camera, the video stream in the above-mentioned shooting preview interface may cause the children to have a little distortion compared to other adults. The actual image of the child is different from the real image, which affects the shooting effect.

Based on this, as a possible implementation manner, on the basis of the foregoing description, for each face image after rotation, a pre-trained age estimation model is used to recognize the face image, and the face image in the face image is obtained. Face age, and judge whether the face age in the face image is greater than the set age, if the face age in the face image is less than the set age, then the face age in the face image The size of the face frame of the face image is corrected.

In detail, as an example, the electronic device may pre-store the median of the face circumference corresponding to the age of each face, and then the face frame of the face image according to the face age in the face image The size correction method can be as follows: First, obtain the first median of the face circumference corresponding to the age of the face in the face image and the second median of the face circumference corresponding to the set age Then, the face frame correction coefficient is calculated according to the first median and the second median, and finally the face frame size of the face image is corrected according to the face frame correction coefficient.

Taking the set age as 18 years old, the face age in the face image is 10 years old as an example, suppose the first median of the face circumference corresponding to 10 years old is b ₁ , and the face circumference corresponding to 18 years old The first median of is b ₂ , then the calculated face frame correction coefficient is b ₁ /b ₂ . On this basis, the method of correcting the size of the face frame of the face image according to the face frame correction coefficient may be: using the face frame correction coefficient b ₁ /b _{2 to} respectively correct the face image of the face image. The side length and area of the face frame are corrected.

For example, the corrected side length L ₁ of the face frame of the face image may be between the original side length L _{0 of the} face frame of the face image and the face frame correction coefficient b ₁ /b ₂ Quotient value, namely L ₁ =L ₀ /(b ₁ /b ₂ );

For another example, the corrected area S ₁ of the face frame of the face image may be the square value of the original area S _{0 of the} face frame of the face image and the face frame correction coefficient b ₁ /b ₂ The quotient between S ₁ =S ₀ /(b ₁ /b ₂ ) ² .

It is worth noting that the median face circumference corresponding to the age of each face can be obtained by collecting a large number of samples of different ages. The specific data can be fine-tuned according to actual needs, which is not specifically limited in this embodiment.

In this way, the degree of distortion that children may have in the video stream in the above-mentioned shooting preview interface compared to other adults can be reduced.

Further, for step S120, as a possible implementation manner, according to the face frame information corresponding to each face, the face with the largest face frame area may be determined as the face to be corrected, and the face to be corrected may be calculated The relative distance coefficient between the face and the camera lens.

For example, according to the face frame information corresponding to each face, the average area of the face frame corresponding to all recognized faces can be calculated, and then the area of the face frame corresponding to each face can be calculated as compared with the average area. And finally calculate the relative distance coefficient between the face to be corrected and the camera lens according to the sum of the square difference values. In this example, the specific calculation formula of the relative distance coefficient can be:

Where, d is the relative distance coefficient between the face to be corrected and the camera lens, N is the number of faces, x _i is the area of the ith face frame, and r is the average area of the face frame corresponding to all faces.

For another example, it is also possible to obtain the median area and the maximum number area corresponding to the face frame of the face other than the face to be corrected, and calculate the median area and the person whose face is to be corrected The first ratio of the face frame area and the second ratio of the maximum number area to the face frame area of the face to be corrected, and finally according to the weight coefficients corresponding to the preset first and second ratios, the The first ratio and the second ratio calculate a relative distance coefficient between the face to be corrected and the camera lens. In this example, the specific calculation formula of the relative distance coefficient can be:

Further, for step S130, the electronic device may pre-store a plurality of distortion correction parameters corresponding to preset distance coefficients, so that when the relative distance coefficient obtained in step S120 is a preset distance coefficient stored in advance, it can be directly obtained The distortion correction parameter corresponding to the relative distance coefficient. However, in a real scene, the relative distance coefficient obtained in step S120 is usually difficult to match the accurate preset distance coefficient. It is very difficult to collect all the distortion correction parameters corresponding to the preset distance coefficient. In addition, if the If the relative distance coefficient obtained in step S120 is matched with the distortion correction parameter corresponding to the similar preset distance coefficient, there will be a distortion correction error.

For this reason, the inventor proposes the following exemplary solutions to the above problems:

First, obtain a preset distance coefficient range in which the relative distance coefficient is located. The preset distance coefficient range may include a first endpoint value and a second endpoint value, and the first endpoint value is smaller than the second endpoint value. .

Next, calculate the first difference between the relative distance coefficient and the first endpoint value, the second difference between the second endpoint value and the first endpoint value, and the corresponding value of the second endpoint The third difference between the distortion correction parameter and the distortion correction parameter corresponding to the first endpoint value.

Finally, the corresponding target distortion correction parameter is calculated according to the first endpoint value, the first difference value, the second difference value, and the third difference value.

Wherein, the target distortion correction parameter can be obtained by the following calculation formula:

For example, assuming that the distortion correction parameter corresponding to the preset distance coefficient of 0.15 is 0.3, the distortion correction parameter corresponding to the preset distance coefficient of 0.20 is 0.5, and the relative distance coefficient calculated by the above step S120 is 0.17, then the target distortion correction parameter is:

In this way, through the above solution, more accurate target distortion correction parameters can be obtained, thereby reducing the error of the distortion correction value, and there is no need to spend a lot of time and cost to collect all the distortion correction parameters corresponding to the preset distance coefficients.

Based on the foregoing, the human face to be corrected may be subjected to distortion correction according to the target distortion correction parameter to obtain a target image after distortion correction.

As an example, first establish the face grid of the face to be corrected, determine each constraint point in the face grid, and then calculate the face grid in the face grid according to the target distortion correction parameter The constraint deformation variables of each constraint point are then adjusted according to the calculated constraint deformation variables of each constraint point to obtain the adjusted face grid, and the face to be corrected is mapped to the adjustment After the face grid, the target image after distortion correction is obtained.

Further, referring to FIG. 3, after step S130, the image distortion correction method provided in this embodiment may further include the following steps:

Step S140: Display each target image after distortion correction in the shooting preview interface in real time, and when a shooting instruction is detected, the target image currently displayed in the shooting preview interface is taken as the shooting image and stored in the shooting preview interface. Mentioned in electronic equipment.

In this embodiment, each target image after distortion correction is displayed in the shooting preview interface in real time, so that the photographer can see the target image after distortion correction instead of the target image that has been distorted before taking a picture. Correct the image. Later, when you need to take a picture, you can press the shooting button, and when a shooting instruction is detected, the target image currently displayed in the shooting preview interface is used as the shooting image and stored in the electronic device.

In other words, in this embodiment, face recognition can be performed on the image to be corrected first to obtain face frame information and face key points corresponding to at least two faces in the image to be corrected. Then, according to the face frame information corresponding to each recognized face, the face to be corrected in the image to be corrected is determined, and the relative distance coefficient between the face to be corrected and the camera lens is calculated. Then, distortion correction is performed on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction.

Optionally, since in a multi-person selfie scene, usually the user holding the camera is closest to the camera, the face frame area corresponding to the user is the largest, and the face image of the user is the image that needs to be corrected. Therefore, in this embodiment, it is possible to determine the face with the largest face frame area as the face to be corrected according to the face frame information corresponding to each face, and then calculate the relative distance between the face to be corrected and the camera lens coefficient.

Optionally, when correcting the face to be corrected, the face to be corrected needs to be corrected to the size of the other faces in the image to be corrected. Therefore, in the embodiment, it can be calculated according to the area of the face frame of each face The relative distance coefficient between the face to be corrected and the camera. For the specific calculation method of the distance coefficient, please refer to the preceding text, which will not be repeated here.

Optionally, since the face of a child is usually smaller than that of an adult, the area of the child’s face image will be smaller than the area of the adult’s face image at the same distance, which may lead to subsequent corrections based on the face image area. When calculating face recognition or relative distance coefficient, the calculation result is not accurate. Therefore, in this embodiment, the face age corresponding to each face image in the image to be corrected can be identified first, and the face frame size of each face image is corrected according to the face age corresponding to each face image, and then Then perform the determination of the face to be corrected or the calculation of the relative distance coefficient. In this way, the influence of the size of the child's face on the subsequent determination of the face to be corrected and the calculation of the relative distance can be reduced. Among them, please refer to the previous text for specific correction methods, so I won't repeat them here.

Optionally, since the skew of the face image being photographed when self-calculating the size of the face frame may cause the calculation of the face frame size to be inaccurate, thereby affecting subsequent calculations, in this embodiment, after obtaining the to-be-corrected After the face frame information and face key points corresponding to at least two faces in the image, for each face, the corresponding face image is cut out according to the face frame information corresponding to the face; according to the person corresponding to the face Face key points, using an affine matrix to rotate the face image to a set position. In this way, the calculation results of the subsequent calculation steps based on the size of the face frame can be made more accurate.

In order to more clearly describe the beneficial effects of the image distortion correction method of the present application, the following examples are given in conjunction with the implementation process in specific application scenarios. Examples are as follows:

As an example, in a multi-person photo scene in daily life, such as taking a group photo of people while traveling, usually the photographer holds the electronic device while the rest of the people choose a suitable location to take the photo. As shown in Figure 4, after the photographer opens the camera application of the electronic device, the electronic device turns on the camera B4 and displays the shooting preview interface L. The shooting preview interface L displays the real-time video stream of the group photo in the travel scene obtained by the camera B4 , The photographer can also select the shooting mode button B2 to choose front camera or rear camera, but whether it is front camera or rear camera, the photographer is the person closest to the camera B4, which will cause the group to take a selfie In the scene, the face of the photographer (the face to be corrected) Face0 will be distorted. It can be seen from Figure 4 that this distortion will be more obvious when a group photo is taken.

If the image distortion correction method provided in the embodiment of the present application is adopted, as shown in FIG. 5, the distortion of the face of the photographer's face after correction is obviously greatly improved. Therefore, each time the shooting preview interface L displays the video stream after distortion correction, thereby optimizing the shooting effect in the multi-person selfie photo scene.

On this basis, after the photographer presses the shooting button B1, the image currently displayed in the shooting preview interface L can be used as a shooting image and stored, and the just stored shooting image can also be viewed through the shooting image preview frame B3.

FIG. 6 shows a schematic diagram of an electronic device 100 provided by an embodiment of the present application. In this embodiment, the electronic device 100 may include a storage medium 110, a processor 120, and an image distortion correction device 130.

The processor 120 may be a general-purpose central processing unit (Central Processing Unit, CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more for controlling the implementation of the foregoing methods The example provides an integrated circuit for program execution of the image distortion correction method.

The storage medium 110 may be ROM or other types of static storage devices that can store static information and instructions, RAM or other types of dynamic storage devices that can store information and instructions, or may be an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read Only Memory). Programmable-Only Memory, EEPROM), CD-ROM (Compact disc Read-Only Memory, CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disks A storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The storage medium 110 may exist independently and is connected to the processor 120 through a communication bus. The storage medium 110 may also be integrated with the processor. The storage medium 110 is configured to store application program codes for executing the solution of the present application, such as the image distortion correction device 130 shown in FIG. 5, and is controlled by the processor 120 to execute. The processor 120 is configured to execute application program codes stored in the storage medium 110, such as the image distortion correction device 130, to execute the image distortion correction method of the foregoing method embodiment.

The present application may divide the image distortion correction device 130 into functional modules according to the foregoing method embodiments. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in this application is illustrative and only a logical function division, and there may be other division methods in actual implementation. For example, in the case of dividing each functional module corresponding to each function, the image distortion correction device 130 shown in FIG. 6 is only a schematic diagram of the device. For example, the image distortion correction device 130 shown in FIG. 6 may include an identification module 131, a calculation module 132, and a distortion correction module 133. The functions of each functional module of the image distortion correction device 130 are respectively described in detail below.

The recognition module 131 is configured to perform face recognition on the image to be corrected to obtain face frame information and face key points corresponding to at least two human faces in the image to be corrected. It can be understood that the identification module 131 may be configured to execute the above step S110, and for the detailed implementation of the identification module 131, please refer to the content related to the above step S110.

The calculation module 132 is configured to determine the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and to calculate the relative distance coefficient between the face to be corrected and the camera lens. It can be understood that the calculation module 132 may be configured to execute the foregoing step S120, and for the detailed implementation of the calculation module 132, refer to the foregoing content related to the step S120.

The distortion correction module 133 is configured to perform distortion correction on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction. It can be understood that the distortion correction module 133 may be configured to execute the above-mentioned step S130, and for the detailed implementation of the distortion correction module 133, please refer to the above-mentioned content related to the step S130.

Optionally, referring to FIG. 7, the image distortion correction device 130 may further include:

The display storage module 134 is configured to display each target image after distortion correction in the shooting preview interface in real time, and when a shooting instruction is detected, use the target image currently displayed in the shooting preview interface as the shooting image And stored in the electronic device 100. It can be understood that the display storage module 134 can be configured to execute the above step S140, and for the detailed implementation of the display storage module 134, please refer to the content related to the above step S140.

Since the image distortion correction device 130 provided by the embodiment of the present application is another implementation form of the image distortion correction method shown in FIG. 1 or FIG. 3, and the image distortion correction device 130 can be configured to execute the image distortion correction method shown in FIG. 1 or FIG. The image distortion correction method provided by the embodiment, therefore, the technical effect that can be obtained can refer to the above method embodiment, and it will not be repeated here.

Optionally, based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the above-mentioned image distortion correction method A step of.

Specifically, the storage medium can be a general storage medium, such as a portable disk, a hard disk, etc., and when the computer program on the storage medium is executed, the above-mentioned image distortion correction method can be executed.

The embodiments of the present application are described with reference to the flowcharts and/or block diagrams of the method, the device (the electronic device 100 shown in FIG. 6 or FIG. 7), and the computer program product according to the embodiments of the present application. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

Although the present application is described with reference to various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and understand by viewing the drawings, the disclosure, and the appended claims. Implement other changes of the disclosed embodiment. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit may implement several functions listed in the claims. Certain measures are described in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

The above are only the various embodiments of the application, but the protection scope of the application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the application. All should be covered within the scope of protection of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Industrial applicability

The embodiment of this application obtains the face frame information and face key points corresponding to at least two faces in the image to be corrected by performing face recognition on the image to be corrected, and then determines the face frame information corresponding to each recognized face. Correct the face to be corrected in the image, calculate the relative distance coefficient between the face to be corrected and the camera lens, and then perform distortion correction on the face to be corrected according to the relative distance coefficient to obtain the target image after distortion correction. In this way, it is possible to automatically recognize the face to be corrected for distortion correction in real-time when multiple people take a selfie together, thereby optimizing the shooting effect in a scene where multiple people take a selfie together.

Claims

An image distortion correction method, characterized in that it is applied to an electronic device, and the method includes:

Performing face recognition on the image to be corrected to obtain face frame information and face key points corresponding to at least two faces in the image to be corrected;

Determine the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculate the relative distance coefficient between the face to be corrected and the camera lens;

Distortion correction is performed on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction.
The image distortion correction method according to claim 1, wherein the step of performing face recognition on the image to be corrected to obtain face frame information and face key points corresponding to at least two faces in the image to be corrected ,include:

After detecting the camera start command, turn on the camera and enter the shooting preview interface;

For each frame of the image to be corrected in the shooting preview interface, face recognition is performed on the frame of the image to be corrected through the face recognition model obtained in advance to obtain the face frame information corresponding to each face in the frame of the image to be corrected And key points of the face;

Wherein, the face recognition model uses multiple training samples and the labeled data of each training sample to be obtained based on deep learning neural network training, where the labeled data of each training sample includes the face frame corresponding to each face in the training sample Key points of information and faces.
The image distortion correction method according to claim 1 or 2, wherein the face frame information corresponding to each recognized face is used to determine the face to be corrected in the image to be corrected, and calculate the Before the step of correcting the relative distance coefficient between the face and the camera lens, the method further includes:

For each face, crop out the corresponding face image according to the face frame information corresponding to the face;

According to the key points of the face corresponding to the face, the face image is rotated to the set position using the affine matrix.
4. The image distortion correction method according to claim 3, wherein the method further comprises:

For each face image after rotation, a pre-trained age estimation model is used to recognize the face image to obtain the face age in the face image;

Determine whether the age of the face in the face image is greater than the set age;

If the age of the face in the face image is less than the set age, the size of the face frame of the face image is corrected according to the age of the face in the face image.
The image distortion correction method according to claim 4, wherein the electronic device pre-stores the median of the face circumference corresponding to the age of each face, and the face image in the face image The steps to correct the size of the face frame of the face image by age include:

Acquiring the first median of the face circumference corresponding to the age of the face in the face image and the second median of the face circumference corresponding to the set age;

Calculating a face frame correction coefficient according to the first median and the second median;

The size of the face frame of the face image is corrected according to the face frame correction coefficient.
The image distortion correction method according to any one of claims 1 to 5, wherein said determining the face to be corrected in the image to be corrected according to face frame information corresponding to each recognized face, And the step of calculating the relative distance coefficient between the face to be corrected and the camera lens includes:

According to the face frame information corresponding to each face, determine the face with the largest face frame area as the face to be corrected;

Calculate the relative distance coefficient between the face to be corrected and the camera lens.
8. The image distortion correction method according to claim 6, wherein the step of calculating the relative distance coefficient between the face to be corrected and the camera lens comprises:

Calculating the average area of the face frame corresponding to all recognized faces according to the face frame information corresponding to each face;

Calculating the sum of the square value of the difference between the area of the face frame corresponding to each face and the average area;

The relative distance coefficient between the face to be corrected and the camera lens is calculated according to the sum of the squared values of the difference. The specific calculation formula is:

Where, d is the relative distance coefficient between the face to be corrected and the camera lens, N is the number of faces, x i x i is the area of the i-th face frame, and r is the average area of the face frame corresponding to all faces.
8. The image distortion correction method according to claim 6, wherein the step of calculating the relative distance coefficient between the face to be corrected and the camera lens comprises:

Acquiring the median area and the maximum number area corresponding to the face frame of the face other than the face to be corrected;

Calculating a first ratio of the median area to the face frame area of the face to be corrected and a second ratio of the maximum number area to the face frame area of the face to be corrected;

The relative distance coefficient between the face to be corrected and the camera lens is calculated according to the weight coefficients corresponding to the preset first ratio and second ratio, the first ratio and the second ratio, and the specific calculation formula is:

Where d is the relative distance coefficient between the face to be corrected and the camera lens, a max is the maximum number area corresponding to the face frame of the face other than the face to be corrected, and a mid is the face to be corrected The median area corresponding to the face frame of the face other than the face, K is a constant between 0 and 1, and y is the face frame area of the face to be repaired.
The image distortion correction method according to any one of claims 1-8, wherein the electronic device pre-stores a plurality of distortion correction parameters corresponding to preset distance coefficients, and the pair of distortion correction parameters according to the relative distance coefficients The step of performing distortion correction on the face to be corrected to obtain a target image after distortion correction includes:

Acquiring a preset distance coefficient range in which the relative distance coefficient is located, the preset distance coefficient range including a first endpoint value and a second endpoint value, the first endpoint value is less than the second endpoint value;

Calculate the first difference between the relative distance coefficient and the first endpoint value, the second difference between the second endpoint value and the first endpoint value, and the distortion correction corresponding to the second endpoint value The third difference between the parameter and the distortion correction parameter corresponding to the first endpoint value;

Calculating corresponding target distortion correction parameters according to the first endpoint value, the first difference value, the second difference value, and the third difference value;

Performing distortion correction on the face to be corrected according to the target distortion correction parameter to obtain a target image after the distortion correction;

Wherein, the target distortion correction parameter is obtained by the following calculation formula:

Where z is the target distortion correction parameter, d is the relative distance coefficient between the face to be corrected and the camera lens, d 1 is the first endpoint value, d 2 is the second endpoint value, and c 1 is the The distortion correction parameter corresponding to the first endpoint value, and c 2 is the distortion correction parameter corresponding to the first endpoint value.
9. The image distortion correction method according to claim 9, wherein the step of performing distortion correction on the face to be corrected according to the target distortion correction parameter to obtain a target image after the distortion correction comprises:

Establishing a face grid of the face to be corrected, and determining each constraint point in the face grid;

Calculating the constraint deformation variables of each constraint point in the face grid according to the target distortion correction parameter;

Adjust the coordinates of each constraint point according to the calculated constraint deformation variables of each constraint point to obtain an adjusted face grid;

The face to be corrected is mapped to the adjusted face grid to obtain a target image after distortion correction.
11. The image distortion correction method according to claims 2-10, wherein the method further comprises:

Display each target image after distortion correction in the shooting preview interface in real time, and when a shooting instruction is detected, use the target image currently displayed in the shooting preview interface as a shooting image and store it in the electronic device in.
The image distortion correction method according to claim 1, wherein the step of determining the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face comprises:

According to the face frame information corresponding to each face, it is determined that the face with the largest face frame area is the face to be corrected.
The image distortion correction method according to claim 12, wherein the step of calculating the relative distance coefficient between the face to be corrected and the camera lens comprises:

The relative distance coefficient between the face to be corrected and the camera is calculated according to the area of the face frame of each face.
The image distortion correction method according to claim 12 or 13, wherein before the determining that the face with the largest face frame area is the face to be corrected, the method further comprises:

Identifying the age of the face corresponding to each face image in the image to be corrected;

The size of the face frame of each face image is corrected according to the face age corresponding to each face image.
The image distortion correction method according to any one of claims 12-14, wherein the face frame information corresponding to each recognized face is determined to determine the face to be corrected in the image to be corrected, and Before the step of calculating the relative distance coefficient between the face to be corrected and the camera lens, the method further includes:

For each face, crop out the corresponding face image according to the face frame information corresponding to the face;

According to the key points of the face corresponding to the face, the face image is rotated to the set position using the affine matrix.
An image distortion correction device, characterized in that it is applied to electronic equipment, and the device includes:

A recognition module configured to perform face recognition on the image to be corrected, and obtain face frame information and face key points corresponding to at least two faces in the image to be corrected;

A calculation module, configured to determine the face to be corrected in the image to be corrected according to the face frame information corresponding to each recognized face, and calculate the relative distance coefficient between the face to be corrected and the camera lens;

The distortion correction module is configured to perform distortion correction on the face to be corrected according to the relative distance coefficient to obtain a target image after distortion correction.
An electronic device, characterized in that the electronic device includes one or more storage media and one or more processors in communication with the storage media, and the one or more storage media stores machine executable instructions executable by the processor When the electronic device is running, the processor executes the machine executable instructions to realize the image distortion correction method according to any one of claims 1-15.
A readable storage medium, wherein the readable storage medium stores machine-executable instructions, and the machine-executable instructions realize the image distortion correction method according to any one of claims 1-15 when executed .