CN114125273B

CN114125273B - Face focusing method and device and electronic equipment

Info

Publication number: CN114125273B
Application number: CN202111306696.7A
Authority: CN
Inventors: 陈典浩
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2021-11-05
Filing date: 2021-11-05
Publication date: 2023-04-07
Anticipated expiration: 2041-11-05
Also published as: CN114125273A

Abstract

The application discloses a face focusing method, a face focusing device and electronic equipment, and belongs to the technical field of camera shooting. The face focusing method provided by the application comprises the following steps: inputting the target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model; focusing the human face based on the target focal length; the target parameter detection model is obtained based on two-dimensional face image samples, a combined three-dimensional face deformation model and a camera model; the three-dimensional face deformation model is used for obtaining a three-dimensional face based on the input face characteristic parameters; the camera model is used for obtaining a restored two-dimensional face image based on the input pose parameters and the three-dimensional face; the face characteristic parameters and the pose parameters are parameters output by an initial parameter detection model based on a two-dimensional face image sample; the pose parameters are parameters representing shooting postures of the camera; the face feature parameters are parameters for representing the face contour.

Description

Face focusing method and device and electronic equipment

Technical Field

The application belongs to the technical field of camera shooting, and particularly relates to a face focusing method and device and electronic equipment.

Background

In real life, when a user uses an electronic device to take a picture, if a person needs to be taken, the user usually needs to focus on a face of the person.

In the related technology, when the face is focused, firstly, the face detection is carried out, and an interested area is obtained according to a face detection frame; then adjusting the region of interest according to the pose of the face and the position of the face in the image; then, phase image statistics is carried out in the adjusted interested region, and a focusing position is calculated based on a Phase Detection Auto Focus (PDAF) technology and the counted Phase image; and finally, moving the position of the motor according to the focusing position to further finish the automatic focusing of the human face.

However, the adjusted region of interest not only contains face information but also contains background information, the background and the face are not in the same plane, and if the focusing position of the face is calculated based on the background information and the face information, the face focusing accuracy is poor.

Disclosure of Invention

The embodiment of the application aims to provide a face focusing method, a face focusing device and electronic equipment, and the problem that face focusing accuracy is poor can be solved.

In a first aspect, an embodiment of the present application provides a face focusing method, where the method includes:

inputting a target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model;

focusing the human face based on the target focal length;

the target parameter detection model is obtained based on two-dimensional face image samples, a combined three-dimensional face deformation model and a camera model; the three-dimensional face deformation model is used for obtaining a three-dimensional face based on the input face characteristic parameters; the camera model is used for obtaining a restored two-dimensional face image based on the input pose parameters and the three-dimensional face;

the face feature parameters and the pose parameters are parameters output by an initial parameter detection model based on the two-dimensional face image samples; the pose parameters are parameters representing shooting postures of the camera; the face characteristic parameters are parameters for representing the face contour.

In a second aspect, an embodiment of the present application provides a face focusing device, where the face focusing device includes:

the first detection module is used for inputting a target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model;

the focusing module is used for focusing the human face based on the target focal length;

In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and when executed by the processor, the program or instructions implement the steps of the method according to the first aspect.

In a fourth aspect, embodiments of the present application provide a readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the method according to the first aspect.

In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the method according to the first aspect.

In the embodiment of the application, the face is focused through the target focal length obtained by extracting the characteristics of the target image based on the target parameter detection model, the target parameter detection model is obtained based on the training of the face characteristic parameters and the pose parameters and does not relate to background information, the calculation of the target focal length based on the face information is realized, and the face focusing precision is improved.

Drawings

Fig. 1 is a schematic flow chart of a face focusing method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a monocular three-dimensional face reconstruction system according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a deep convolutional neural network provided in an embodiment of the present application;

fig. 4 is a second schematic flowchart of a face focusing method according to an embodiment of the present application;

fig. 5 is a third schematic flowchart of a face focusing method according to an embodiment of the present application;

fig. 6a is one of focus finding principle diagrams of the PDAF technology provided by the embodiment of the present application;

fig. 6b is a second schematic view illustrating a focus searching principle of the PDAF technology according to the embodiment of the present application;

fig. 7a is a third schematic view illustrating a focus finding principle of the PDAF technology according to the embodiment of the present application;

fig. 7b is a fourth schematic view illustrating a focus finding principle of the PDAF technology according to the embodiment of the present application;

fig. 8a is a fifth schematic view of a focus finding principle of the PDAF technology provided in the embodiment of the present application;

fig. 8b is a sixth schematic view illustrating a focus finding principle of the PDAF technology according to the embodiment of the present application;

fig. 9 is a fourth schematic flowchart of a face focusing method according to an embodiment of the present application;

fig. 10 is a schematic flowchart of a shooting method provided in an embodiment of the present application;

fig. 11 is a schematic structural diagram of a face focusing device according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

fig. 13 is a hardware schematic diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present disclosure.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.

The following describes in detail a face focusing method, a face focusing device, and an electronic device according to embodiments of the present application with reference to the accompanying drawings.

In the face focusing method provided in the embodiment of the present application, an execution subject of the face focusing method may be an electronic device or a functional module or a functional entity capable of implementing the face focusing method in the electronic device, the electronic device mentioned in the embodiment of the present application includes, but is not limited to, a mobile phone, a tablet computer, a camera, a wearable device, and the like, and the face focusing method provided in the embodiment of the present application is described below with the electronic device as the execution subject.

Fig. 1 is a schematic flow diagram of a face focusing method according to an embodiment of the present application, and as shown in fig. 1, the face focusing method includes steps 101 and 102:

step 101, inputting a target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model.

The target parameter detection model is obtained based on two-dimensional face image samples, a combined three-dimensional face deformation model (3D Mobile model,3 DMM) and camera model training.

The three-dimensional face deformation model is used for obtaining a three-dimensional face based on the input face characteristic parameters; the camera model is used for obtaining a restored two-dimensional face image based on the input pose parameters and the three-dimensional face; the face characteristic parameters and the pose parameters are parameters output by the initial parameter detection model based on two-dimensional face image samples; the pose parameters are parameters representing shooting postures of the camera; the face feature parameters are parameters for representing the face contour.

Optionally, the three-dimensional face deformation model and the camera model are contents included in a monocular three-dimensional face reconstruction technology, and the monocular three-dimensional face reconstruction technology reconstructs a corresponding three-dimensional face from a single two-dimensional face image. In the real world, the face information is three-dimensional, the two-dimensional face image is the projection of the three-dimensional face to the camera plane, in the projection process, the depth information is lost, and the monocular three-dimensional face reconstruction technology can reconstruct the depth information of the face according to a single two-dimensional face image and reconstruct the three-dimensional face corresponding to the two-dimensional face image.

The three-dimensional face deformation model is a three-dimensional face standard model formed by vertexes and triangular patches, the positions of all the vertexes determine the shape of a face, and the colors of all the vertexes determine the texture of the face; the triangular patch describes the topological relation between the vertexes; the camera model projects a three-dimensional face into a two-dimensional face image.

Fig. 2 is a schematic diagram of a monocular three-dimensional face reconstruction system according to an embodiment of the present application, and as shown in fig. 2, the monocular three-dimensional face reconstruction technology mainly includes a three-dimensional face deformation model, a camera model, and parameter optimization, where the three-dimensional face deformation model inputs a three-dimensional face reconstructed based on a two-dimensional face image sample into the camera model, the camera model projects the three-dimensional face into a restored two-dimensional face image, and the parameter optimization is performed based on the restored two-dimensional face image and the input two-dimensional face image sample.

Optionally, after the target parameter detection model is obtained through training, the obtained target image is input into the pre-trained target parameter detection model, the convolution layer of the target parameter detection model extracts two-dimensional image features from the target image, and finally the target focal length is output through the full connection layer of the target parameter detection model.

It should be noted that, the target parameter detection model may output a target rotation parameter, a target translation parameter, a target shape parameter, and a target texture parameter in addition to the target focal length, and the output target focal length, the target rotation parameter, the target translation parameter, the target shape parameter, and the target texture parameter may be applied to scenes such as face beautifying and intelligent face pinching, which is not limited in this application.

And step 102, focusing the human face based on the target focal length.

Optionally, after the target focal length is obtained, the target focal length may be converted into a corresponding target focusing image distance, and the face is focused based on the target focusing image distance.

According to the face focusing method provided by the embodiment of the application, the face is focused through the target focal length obtained by extracting the characteristics of the target image based on the target parameter detection model, the target parameter detection model is obtained based on the face characteristic parameters and the pose parameters, background information is not involved, the calculation of the target focal length based on the face information is realized, and the face focusing precision is improved.

Optionally, the pose parameters include a focus sample, a rotation parameter sample, and a translation parameter sample; the face feature parameters comprise shape parameter samples and texture parameter samples.

In an embodiment, the rotation parameter samples include three-dimensional rotation parameters, the translation parameter samples include three-dimensional translation parameters, the shape parameter samples include 20-dimensional shape parameters, and the texture parameter samples include 20-dimensional texture parameters, so long as the three-dimensional rotation parameters, the three-dimensional translation parameters, the 20-dimensional shape parameters, the 20-dimensional texture parameters, and the target focal length are estimated, the processes of reconstructing the three-dimensional face and projecting the three-dimensional face into a restored two-dimensional face image can be realized, and further, the optimization of the model parameters of the initial parameter detection model can be realized.

According to the face focusing method provided by the embodiment of the application, the pose parameters comprise the rotation parameter samples, the translation parameter samples and the focal length samples, the face characteristic parameters comprise the shape parameter samples and the texture parameter samples, and the related parameters can correctly reflect the main characteristics of the image, so that the accuracy of the target parameter detection model obtained through final training can be improved.

The following describes the training and optimization process of the target parameter detection model:

in one embodiment, a data set of a target number (e.g., 200) of three-dimensional human faces is acquired, and a human face shape basis vector S is extracted from the data set by a principal component analysis method _i And face texture basis vector T _i . The shape and texture of each three-dimensional face can be linearly represented by the following face shape basis vector S in formula (1) and face texture basis vector T in formula (2):

wherein the content of the first and second substances,

represents an average three-dimensional face shape, <' > based on a human face image>

Representing the average three-dimensional face texture, α _i Base vector S representing ith individual face shape _i Corresponding parameter, β _i Base vector T representing ith personal face texture _i And i is a positive integer corresponding to the parameter.

From equations (1) and (2), it can be derived that all α are estimated _i And beta _i And then the three-dimensional face corresponding to the two-dimensional face image can be reconstructed. For example, a 20-dimensional α parameter and a 20-dimensional β parameter are estimated to reconstruct a three-dimensional face corresponding to the two-dimensional face image.

Further, in order to evaluate the quality of the reconstructed three-dimensional face, the reconstructed three-dimensional face needs to be projected through a camera model to obtain a restored two-dimensional face image.

Suppose that the coordinates of the jth vertex of a three-dimensional face in three-dimensional space are (x) _j ，y _j ，z _j ) After the projection of the camera model, the coordinates of the jth vertex in the two-dimensional space are (u) _j ，v _j ) Specifically, the following formula (3) is used for projection.

Where dx represents a length unit occupied by one pixel in the x-axis direction, dy represents a length unit occupied by one pixel in the y-axis direction, and u ₀ A number of horizontal pixels, v, representing a phase difference between a central pixel coordinate of the two-dimensional face image and an origin pixel coordinate of the two-dimensional face image ₀ The method comprises the steps of representing the number of longitudinal pixels of a phase difference between a central pixel coordinate of a two-dimensional face image and an original point pixel coordinate of the two-dimensional face image, Z representing a manually set imaging plane position, R representing a three-dimensional rotation parameter, t representing a three-dimensional translation parameter, and f representing a focal length.

It can be obtained from the formula (3), and the reconstructed three-dimensional face can be re-projected into a two-dimensional face image as long as the three-dimensional rotation parameter R, the three-dimensional translation parameter t and the focal length f are estimated.

It can be understood that the parameters to be estimated in the three-dimensional face reconstruction process are as follows: the first number-dimensional α parameter, the second number-dimensional β parameter, the three-dimensional rotation parameter R, the three-dimensional translation parameter t, and the focal length f, take the 20-dimensional α parameter and the 20-dimensional β parameter as examples.

In one embodiment, the specific training step of the target parameter detection model includes:

inputting a two-dimensional face image sample into an initial parameter detection model to obtain face characteristic parameters and pose parameters output by the initial parameter detection model; inputting the face characteristic parameters into a three-dimensional face deformation model to obtain a three-dimensional face output by the three-dimensional face deformation model; inputting the pose parameters and the three-dimensional face into a camera model to obtain a restored two-dimensional face image output by the camera model; and determining a target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample.

The initial parameter detection model may be a constructed deep convolutional neural network, and the deep convolutional neural network is used for model training and parameter estimation, fig. 3 is a schematic structural diagram of the deep convolutional neural network provided in the embodiment of the present application, and as shown in fig. 3, the deep convolutional neural network sequentially includes a convolutional layer, a normalization layer, an activation unit layer, an average pooling layer, and a full connection layer.

Inputting a two-dimensional face image sample into a deep convolutional neural network, and extracting two-dimensional image features from the two-dimensional face image sample by a convolutional layer of the deep convolutional neural network; inputting the extracted two-dimensional image features into a normalization layer for normalization processing, and accelerating the convergence speed of the deep convolution neural network; inputting the two-dimensional image features subjected to normalization processing into an activation unit layer for nonlinear transformation; inputting the two-dimensional image features subjected to nonlinear transformation into an average pooling layer, wherein the average pooling layer is used for reducing the size of the input two-dimensional image feature map; and finally, inputting the reduced two-dimensional image characteristic diagram into a full connection layer, wherein the full connection layer is used for re-assembling all the previous local characteristics through a weight matrix, and finally outputting a three-dimensional rotation parameter, a three-dimensional translation parameter, a focal length, a 20-dimensional shape parameter and a 20-dimensional texture parameter.

After the three-dimensional rotation parameter, the three-dimensional translation parameter, the focal length, the 20-dimensional shape parameter and the 20-dimensional texture parameter are obtained, the 20-dimensional shape parameter and the 20-dimensional texture parameter can be input into a three-dimensional face deformation model to reconstruct a three-dimensional face; the reconstructed three-dimensional face can be re-projected into a restored two-dimensional face image based on the three-dimensional rotation parameter, the three-dimensional translation parameter and the focal length.

It should be noted that, in the case that the target image is an image directly acquired in the current scene, the two-dimensional face image sample is also an image directly acquired; in the case where the target image is an image captured based on the initial focus position determined by the PDAF technique, the two-dimensional face image sample is also an image captured based on the initial focus position determined by the PDAF technique.

It should be noted that the reconstructed three-dimensional face may also implement an Augmented Reality (AR) and a Virtual Reality (VR) technique, and the like, which is not limited in this application.

According to the face focusing method provided by the embodiment of the application, the target parameter detection model is determined based on the face characteristic parameters and the pose parameters output by the initial parameter detection model, and the face characteristic parameters and the pose parameters can correctly reflect the main characteristics of an image, so that the accuracy of the target parameter detection model can be improved.

Further, determining a target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample can be specifically realized by the following method:

determining a loss function based on the similarity of the restored two-dimensional face image and the two-dimensional face image sample; and optimizing the model parameters of the initial parameter detection model based on the loss function until the convergence condition is met to obtain the target parameter detection model.

In an embodiment, in order to optimize the model parameters of the model, a loss function based on the restored two-dimensional face image and the two-dimensional face image sample is established, and the loss function reflects the similarity between the restored two-dimensional face image and the two-dimensional face image sample, so that the model parameters of the model are optimized until a convergence condition is met, and a finally optimized target parameter detection model is obtained.

In one embodiment, the Loss function Loss is constructed using the following equation (4) _rec The model parameters of the deep convolutional neural network are optimized.

Loss _rec ＝∑|I _input -I _rec | (4)

Wherein, I _input Representing an input two-dimensional face image sample, I _rec Representing the restored two-dimensional face image.

According to the face focusing method provided by the embodiment of the application, the model parameters of the model are optimized through the loss function, the finally optimized target parameter detection model is obtained, and the accuracy of the target parameter detection model is improved.

Optionally, fig. 4 is a second flowchart of the face focusing method provided in the embodiment of the present application, and as shown in fig. 4, before executing step 101 in fig. 1, the method further includes the following steps:

and 103, acquiring a target image under the conditions of face detection and automatic focusing.

Optionally, when focusing is started, determining whether a face exists in the current scene according to a result returned by the face detection algorithm; if the face detection algorithm has a return value, determining that a face exists in the current scene, and determining that the face exists in the current scene as a portrait scene; if the face detection algorithm does not have a return value, determining that the face does not exist in the current scene and is not the portrait scene, and calling focusing strategies corresponding to other scenes for focusing.

Further, when the scene is determined to be a portrait scene, whether a user operation is received or not is detected, wherein the user operation can be an operation which is performed by a user on a screen and is related to focusing, and when the user operation is received, the user wants to focus on an imaging object at a user operation position, and a touch focusing strategy is called for focusing.

In a portrait scene, when user operation is not received, it can be determined that a user wants the electronic device to perform automatic focusing, and a portrait exists in the current scene, the user usually wants to be able to focus on a face of a person, so that the face of the person is imaged clearly, and at this time, the face automatic focusing strategy is triggered, that is, the electronic device starts to execute a step of acquiring a target image.

It should be noted that the target image acquired by the electronic device may be an image directly acquired in a current scene, or may also be an image acquired at an initial focusing position determined based on a PDAF technology, where the PDAF technology is a focusing method that calculates a focusing position by using Phase Detection (PD) pixel points on an image sensor, and the application is not limited thereto.

According to the face focusing method provided by the embodiment of the application, the target image is obtained only when the face is detected and the face is automatically focused, so that the false triggering of the face focusing method is avoided.

Optionally, fig. 5 is a third schematic flow chart of the face focusing method provided in the embodiment of the present application, and as shown in fig. 5, the implementation manner of step 103 in fig. 4 may include the following steps:

and step 1031, determining an initial focusing position based on the target face detection region, the first phase diagram and the second phase diagram.

The target face detection area is an automatic focusing area, and the first phase diagram and the second phase diagram are obtained according to PD pixels on the image sensor.

In an embodiment, after the face auto-focusing strategy is triggered, the input required by the face auto-focusing strategy needs to be obtained, which mainly includes a target face detection area output by a face detection algorithm, and the target face detection area can be an area in a target face detection frame, namely an auto-focusing area; and a first phase map and a second phase map obtained according to the PD pixels on the image sensor, wherein the first phase map can be a left phase map, and the second phase map can be a right phase map.

And when the target face detection area, the first phase diagram and the second phase diagram are obtained, determining an initial focusing position based on the PDAF technology, the target face detection area, the first phase diagram and the second phase diagram.

The focusing principle of the PDAF technology is as follows: respectively obtaining a first phase diagram and a second phase diagram through PD pixels on the image sensor, and when the phase difference of the first phase diagram and the second phase diagram is 0, the first phase diagram and the second phase diagram are just at the focusing position; when the first phase map and the second phase map have phase difference, the first phase map and the second phase map are in the out-of-focus position, so that the distance between the current position and the in-focus position can be directly calculated through the phase difference of the first phase map and the second phase map.

Fig. 6a is a schematic view of a focusing principle of the PDAF technology provided in the embodiment of the present application, and fig. 6b is a schematic view of a focusing principle of the PDAF technology provided in the embodiment of the present application, where as shown in fig. 6a and fig. 6b, the first phase map and the second phase map have a phase difference and are at an out-of-focus position.

Fig. 7a is a third schematic view of the focusing principle of the PDAF technology provided in the embodiment of the present application, and fig. 7b is a fourth schematic view of the focusing principle of the PDAF technology provided in the embodiment of the present application, as shown in fig. 7a and fig. 7b, the first phase diagram and the second phase diagram are overlapped and located at the in-focus position.

Fig. 8a is a fifth schematic view of a focusing principle of the PDAF technology provided by the embodiment of the present application, and fig. 8b is a sixth schematic view of the focusing principle of the PDAF technology provided by the embodiment of the present application, and as shown in fig. 8a and fig. 8b, the first phase diagram and the second phase diagram also have a phase difference and are located at a defocus position.

Specifically, the calculation process for determining the initial in-focus position based on the first phase map and the second phase map is as follows:

determining the initial focusing position by a traversal method, and assuming that the first phase diagram is PD _left The second phase diagram is PD _right The Shift parameter represents the first phase diagram PD _left The number of pixels shifted, for example, shift =1, denotes the first phase map PD _left Integral second phase diagram PD _right Is shifted by one pixel, shift = -1 denotes the first phase diagram PD _left Integral deviation from the second phase diagram PD _right In practical application, a specific value of the Shift parameter is set, and if the Shift parameter is an integer from-16 to 16, the Shift parameter needs to be traversed from-16 to 16, and the first phase map PD is obtained _left Shift shift _k One pixel value is obtained

Calculate each shift _k Down, moved->

And a second phase diagram PD _right Similarity SAD in target face detection region _k Specific similarity SAD _k Can be expressed by the following formula (5):

wherein, the ROI represents a target face detection Region, i.e. a Region of Interest (ROI), SAD _k Indicates the degree of similarity corresponding to the k-th Shift, shift _k Indicating the kth Shift, l belongs to ROI, l indicates the l-th pixel within the ROI area.

It can be understood that the smaller the SAD value is, the closer the two corresponding phase maps are in the target face detection area, and the closer the two phase maps are to the in-focus position.

Optionally, three minimum values are selected from all the SAD values obtained

And &>

And->

Corresponding->

Corresponding->

And->

Corresponding to

And then based on->

And &>

Fitting a quadratic curve shown in the following equation (6):

SAD＝a*shift ² +b*shift+c (6)

wherein a, b and c are quadratic curve parameters.

Will be provided with

And &>

Substituting into the formula (6) to obtain the following formula (7); will->

And

substituting into the formula (6) to obtain the following formula (8); will->

And &>

Substituting into equation (6) yields the following equation (9):

based on the formula (7), the formula (8) and the formula (9), the conic parameters a, b and c can be calculated, it can be known that the SAD value of the symmetry axis position of the conic shown in the formula (6) is the minimum, which is the initial focus position, so the shift value of the symmetry axis position of the conic shown in the formula (6) is taken as the phase difference value between the first phase map and the second phase map, and the specific phase difference value pd is expressed by the following formula (10):

and 1032, acquiring an image of the target object based on the initial focusing position to obtain a target image.

In one embodiment, when the initial focusing position is obtained, the phase difference value pd is converted into a motor Code value based on the following formula (11), and the position of the motor is moved based on the motor Code value, so that initial focusing is completed, and the focusing position at this time is already close to a human image, so that the imaging plane can image the human face more clearly, and a target image is obtained.

Code＝pd*DCC (11)

Wherein Code is the motor stroke, and the Defocus Conversion Coefficient (DCC) represents the Coefficient constant for converting the phase difference into the motor Code, and is obtained by calibration of a module factory.

According to the face focusing method provided by the embodiment of the application, the initial focusing position is determined based on the PDAF technology, the focus searching is performed again on the basis of the initial focusing position, the finally obtained target focal distance can be more accurate due to the focus searching in two stages, and the face focusing accuracy is improved.

Fig. 9 is a fourth schematic flowchart of a flow chart of a face focusing method provided in the embodiment of the present application, and as shown in fig. 9, the implementation manner of step 102 in fig. 1 may include the following steps:

1021. and determining the target focusing image distance based on the target focal length, the lens focal length and the target image distance.

And the target image distance is the distance from the imaging plane to the lens.

In one embodiment, the target in-focus image distance is calculated using the imaging gaussian formula shown in equation (12) below:

wherein f represents the target focal length, Z represents the target image distance, namely the position of an imaging plane set manually, u represents the object distance, f _camera Denotes a lens focal length, and v denotes a target in-focus image distance.

In the above formula (12), the object distance u of the face is not changed during imaging, and the focal length f of the lens of the camera is not changed _camera For the known parameter, Z is also a known parameter, and the target focal length f is calculated by using the above step 102, so the target focal length v can be obtained by solving the following formula (12), that is, the target focal length v can be obtained by using the following formula (13):

and step 1022, focusing the human face based on the target focusing image distance.

Optionally, after the target focusing image distance is determined, the target focusing image distance is determined as a final focusing position, and the human face is focused based on the target focusing image distance.

Further, the target focus image distance can be converted into a new motor Code based on the following formula (14), i.e. the far focus image distance v burned in the electronic device by the module factory _inf Distance v of far focus _inf Corresponding motor code _inf Near focal length v _micr And a near focal length v _micr Corresponding motor code _micr Converting the target focusing image distance v into the required new motor code by linear interpolation _v Finally based on motor code _v And moving the motor to the target focusing image distance v to finish the final focusing of the human face image.

According to the face focusing method provided by the embodiment of the application, the final target focusing image distance is obtained through calculation based on the target focal distance, the lens focal distance and the target image distance, and the rapid conversion from the target focal distance to the target focusing image distance is realized.

According to the face focusing method provided by the embodiment of the application, on the basis of the PDAF technology, the initial focusing position is optimized on the basis of the target parameter detection model, and the interference of background information in the region of interest on the calculation of the focusing position is solved; the problem that the PDAF technology is difficult to process low-detail face textures is solved, the face focusing precision is improved, and faster and more accurate focusing experience is brought to a user in a portrait shooting scene.

Fig. 10 is a schematic flowchart of a shooting method provided in an embodiment of the present application, and as shown in fig. 10, the shooting method includes step 1001, step 1002, step 1003, step 1004, and step 1005:

step 1001, under the condition that the human face is detected and the automatic focusing is performed, a target image is obtained.

Step 1002, inputting the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model.

The target parameter detection model is obtained based on two-dimensional face image samples, a combined three-dimensional face deformation model and a camera model; the three-dimensional face deformation model is used for obtaining a three-dimensional face based on the input face characteristic parameters; the camera model is used for obtaining a restored two-dimensional face image based on the input pose parameters and the three-dimensional face; the face characteristic parameters and the pose parameters are parameters output by the initial parameter detection model based on two-dimensional face image samples; the pose parameters are parameters representing shooting postures of the camera; the face feature parameters are parameters for representing the face contour.

And step 1003, focusing the human face based on the target focal length.

Step 1004, receiving a first input of a user.

The first input is an operation of a user to perform shooting, and the first input may be expressed in at least one of the following manners:

first, the first input may be represented by a touch operation, including but not limited to a click operation or a press operation, etc., that is, a screen is clicked, and shooting of the target object is triggered.

In this embodiment, receiving the first input of the user may be expressed as receiving a click operation or the like of the user on a shooting interface of the electronic device.

Second, the first input may be represented as a physical key input.

In this embodiment, the body of the electronic device is provided with corresponding physical keys, and receives a first input of the user, which may be expressed as receiving a first input of the user pressing the corresponding physical keys; the first input may also be a combined operation of pressing a plurality of physical keys simultaneously.

And third, the first input may be represented as a voice input.

In this embodiment, the electronic device may receive a user voice such as "small V, start shooting", etc., where small V is a wake-up word of the electronic device.

Step 1005, responding to the first input, shooting the target object based on the target focal length, and obtaining a target shooting image.

In an embodiment, when the first input of the user is detected, a shooting instruction is sent to the image sensor when the user needs to shoot at the moment based on the first input, and the image sensor starts imaging to obtain a target shooting image when receiving the shooting instruction.

It can be understood that, in the case that the first input of the user is not received, it indicates that the user is not satisfied with the current focusing effect, at this time, it needs to continuously determine whether the user has a new focusing instruction, and if the user clicks the screen, it indicates that the user wants to focus on the imaging object at the user operation position; or the electronic equipment determines whether the shooting scene is changed by the user at the moment by adopting the face detection algorithm again, and if the shooting scene is determined to be changed by the user, the focusing strategy corresponding to the shooting scene is switched to focus.

According to the shooting method provided by the embodiment of the application, the face is focused through the target focal length obtained by extracting the characteristics of the target image based on the target parameter detection model, the target parameter detection model is obtained based on the training of the face characteristic parameters and the pose parameters and does not relate to background information, the calculation of the target focal length based on the face information is realized, and the face focusing precision is improved; when the target object is shot based on the target focal length, the definition of the obtained shot image can be improved, and the quality of the shot image is also improved.

It should be noted that, in the face focusing method provided in the embodiment of the present application, the execution main body may be a face focusing device, or a control module in the face focusing device for executing the face focusing method. The embodiment of the present application describes a face focusing apparatus, which is provided by the embodiment of the present application, by taking a face focusing method executed by a face focusing apparatus as an example.

The embodiment of the application also provides a face focusing device. Fig. 11 is a schematic structural diagram of a face focusing apparatus provided in an embodiment of the present application, and as shown in fig. 11, the apparatus includes a first detection module 1101 and a focusing module 1102; wherein the content of the first and second substances,

the first detection module 1101 is configured to input a target image into a target parameter detection model, so as to obtain a target focal length output by the target parameter detection model;

a focusing module 1102, configured to focus the face based on the target focal length;

the face feature parameters and the pose parameters are parameters output by an initial parameter detection model based on a two-dimensional face image sample; the pose parameters are parameters representing shooting postures of the camera; the face characteristic parameters are parameters for representing the face contour.

According to the face focusing device provided by the embodiment of the application, the face is focused through the target focal length obtained by extracting the characteristics of the target image based on the target parameter detection model, the target parameter detection model is obtained based on the face characteristic parameters and the pose parameters, background information is not involved, the calculation of the target focal length based on the face information is realized, and the face focusing precision is improved.

Optionally, the apparatus further comprises:

the second detection module is used for inputting the two-dimensional face image sample into the initial parameter detection model to obtain face characteristic parameters and pose parameters output by the initial parameter detection model;

the deformation module is used for inputting the face characteristic parameters into the three-dimensional face deformation model to obtain a three-dimensional face output by the three-dimensional face deformation model;

the restoration module is used for inputting the pose parameters and the three-dimensional face into the camera model to obtain a restored two-dimensional face image output by the camera model;

and the determining module is used for determining a target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample.

Optionally, the determining module is further configured to:

determining a loss function based on the similarity of the restored two-dimensional face image and the two-dimensional face image sample;

and optimizing the model parameters of the initial parameter detection model based on the loss function until the convergence condition is met to obtain the target parameter detection model.

Optionally, the apparatus further comprises:

and the acquisition module is used for acquiring the target image under the conditions of face detection and automatic focusing.

Optionally, the obtaining module is further configured to:

determining an initial focusing position based on a target face detection area, a first phase diagram and a second phase diagram, wherein the target face detection area is an automatic focusing area, and the first phase diagram and the second phase diagram are obtained according to PD pixels on an image sensor;

and acquiring an image of the target object based on the initial focusing position to obtain a target image.

Optionally, the focusing module 1102 is further configured to:

determining a target focusing image distance based on the target focal length, the lens focal length and the target image distance; the target image distance is the distance from the imaging plane to the lens;

and focusing the human face based on the target focusing image distance.

The face focusing device in the embodiment of the present application may be a device, and may also be a component, an integrated circuit, or a chip in a terminal. The device can be mobile electronic equipment or non-mobile electronic equipment. By way of example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine or a self-service machine, and the like, and the embodiments of the present application are not particularly limited.

The face focusing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which is not specifically limited in the embodiment of the present application.

The face focusing device provided in the embodiment of the present application can implement each process implemented in the method embodiments of fig. 1 to fig. 10, and can achieve the same technical effect, and is not described herein again to avoid repetition.

Optionally, as shown in fig. 12, an electronic device 1200 is further provided in an embodiment of the present application, and includes a processor 1201, a memory 1202, and a program or an instruction stored in the memory 1202 and executable on the processor 1201, where the program or the instruction is executed by the processor 1201 to implement each process of the foregoing embodiment of the face focusing method, and can achieve the same technical effect, and no further description is provided here to avoid repetition.

It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 13 is a schematic hardware structure diagram of an electronic device implementing an embodiment of the present application.

The electronic device 1300 includes, but is not limited to: a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, a processor 1310, and the like.

Those skilled in the art will appreciate that the electronic device 1300 may further comprise a power supply (e.g., a battery) for supplying power to the various components, and the power supply may be logically connected to the processor 1310 via a power management system, so as to manage charging, discharging, and power consumption management functions via the power management system. The electronic device structure shown in fig. 13 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.

The processor 1310 is configured to input the target image into the target parameter detection model, and obtain a target focal length output by the target parameter detection model;

a processor 1310, further configured to focus the face based on the target focal distance;

the face feature parameters and the pose parameters are parameters output by an initial parameter detection model based on a two-dimensional face image sample; the pose parameters are parameters representing shooting postures of the camera; the face feature parameters are parameters for representing the face contour.

According to the electronic equipment provided by the embodiment of the application, the face is focused through the target focal length obtained by extracting the characteristics of the target image based on the target parameter detection model, the target parameter detection model is obtained based on the training of the face characteristic parameters and the pose parameters, and does not relate to background information, the calculation of the target focal length based on the face information is realized, and the face focusing precision is improved.

Optionally, the processor 1310 is further configured to input the two-dimensional face image sample into the initial parameter detection model, so as to obtain a face feature parameter and a pose parameter output by the initial parameter detection model;

inputting the face characteristic parameters into a three-dimensional face deformation model to obtain a three-dimensional face output by the three-dimensional face deformation model;

inputting the pose parameters and the three-dimensional face into a camera model to obtain a restored two-dimensional face image output by the camera model;

and determining a target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample.

According to the electronic equipment provided by the embodiment of the application, the target parameter detection model is determined based on the face characteristic parameters and the pose parameters output by the initial parameter detection model, and the face characteristic parameters and the pose parameters can correctly reflect the main characteristics of an image, so that the accuracy of the target parameter detection model can be improved.

Optionally, the processor 1310 is further configured to determine a loss function based on the similarity between the restored two-dimensional face image and the two-dimensional face image sample;

According to the electronic equipment provided by the embodiment of the application, model parameters of the model are optimized through the loss function, a finally optimized target parameter detection model is obtained, and the accuracy of the target parameter detection model is improved.

According to the electronic equipment provided by the embodiment of the application, the pose parameters comprise the rotation parameter sample, the translation parameter sample and the focal length sample, the face characteristic parameters comprise the shape parameter sample and the texture parameter sample, and the related parameters can correctly reflect the main characteristics of an image, so that the accuracy of a target parameter detection model obtained through final training can be improved.

Optionally, the processor 1310 is further configured to acquire a target image in case of detecting a human face and performing auto-focusing.

According to the electronic equipment provided by the embodiment of the application, the target image is acquired only when the face is detected and the focusing is performed automatically, so that the false triggering of the face focusing method is avoided.

Optionally, the processor 1310 is further configured to determine an initial focusing position based on a target face detection region, the first phase map and the second phase map, where the target face detection region is an auto-focusing region, and the first phase map and the second phase map are obtained according to PD pixels on the image sensor;

The electronic equipment provided by the embodiment of the application determines the initial focusing position based on the PDAF technology, re-focusing is performed on the basis of the initial focusing position, the final obtained target focal distance can be more accurate by focusing in two stages, and the face focusing accuracy is improved.

Optionally, the processor 1310 is further configured to determine a target focusing image distance based on the target focal length, the lens focal length, and the target image distance; the target image distance is the distance from the imaging plane to the lens;

and focusing the human face based on the target focusing image distance.

The electronic equipment provided by the embodiment of the application calculates the final target focusing image distance based on the target focal length, the lens focal length and the target image distance, and realizes the rapid conversion from the target focal length to the target focusing image distance.

It should be understood that in the embodiment of the present application, the input Unit 1304 may include a Graphics Processing Unit (GPU) 13041 and a microphone 13042, and the Graphics processor 13041 processes image data of still pictures or videos obtained by an image capturing apparatus (such as a camera) in a video capturing mode or an image capturing mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 1307 includes a touch panel 13071 and other input devices 13072. Touch panel 13071, also known as a touch screen. The touch panel 13071 may include two parts, a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein. Memory 1309 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. The processor 1310 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 1310.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the above-mentioned face focusing method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and so on.

The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the above embodiment of the face focusing method, and can achieve the same technical effect, and the details are not repeated here to avoid repetition.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as a system-on-chip, or a system-on-chip.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A face focusing method is characterized by comprising the following steps:

focusing the face based on the target focal length;

the face feature parameters and the pose parameters are parameters output by an initial parameter detection model based on the two-dimensional face image samples; the pose parameters are parameters representing shooting postures of the camera, and the pose parameters comprise focal length samples; the face characteristic parameters are parameters for representing the face contour;

determining a loss function based on the similarity of the restored two-dimensional face image and the two-dimensional face image sample; and optimizing the model parameters of the initial parameter detection model based on the loss function until a convergence condition is met to obtain the target parameter detection model.

2. The method of claim 1, wherein before the target image is input into the target parameter detection model and the target focal length output by the target parameter detection model is obtained, the method further comprises:

inputting the two-dimensional face image sample into the initial parameter detection model to obtain the face characteristic parameters and the pose parameters output by the initial parameter detection model;

inputting the face characteristic parameters into the three-dimensional face deformation model to obtain the three-dimensional face output by the three-dimensional face deformation model;

and inputting the pose parameters and the three-dimensional face into the camera model to obtain the restored two-dimensional face image output by the camera model.

3. The face focusing method according to claim 1, characterized in that the pose parameters further comprise rotation parameter samples and translation parameter samples; the face feature parameters comprise shape parameter samples and texture parameter samples.

4. The method of claim 1, wherein before the inputting the target image into the target parameter detection model and obtaining the target focal length output by the target parameter detection model, the method further comprises:

and acquiring the target image under the condition of detecting the face and automatically focusing.

5. The face focusing method of claim 4, wherein the acquiring the target image comprises:

and acquiring an image of the target object based on the initial focusing position to obtain the target image.

6. The method of any one of claims 1 to 5, wherein focusing the human face based on the target focal length comprises:

determining a target focusing image distance based on the target focal length, the lens focal length and the target image distance; the target image distance is the distance from an imaging plane to a lens;

and focusing the human face based on the target focusing image distance.

7. A face focusing device, comprising:

the device further comprises:

a determining module, configured to determine a loss function based on a similarity between the restored two-dimensional face image and the two-dimensional face image sample; and optimizing the model parameters of the initial parameter detection model based on the loss function until a convergence condition is met to obtain the target parameter detection model.

8. The face focusing device of claim 7, further comprising:

the second detection module is used for inputting the two-dimensional face image sample into the initial parameter detection model to obtain the face characteristic parameters and the pose parameters output by the initial parameter detection model;

the deformation module is used for inputting the face characteristic parameters into the three-dimensional face deformation model to obtain the three-dimensional face output by the three-dimensional face deformation model;

and the restoration module is used for inputting the pose parameters and the three-dimensional face into the camera model to obtain the restored two-dimensional face image output by the camera model.

9. The face focusing device according to claim 7, characterized in that the pose parameters further comprise rotation parameter samples and translation parameter samples; the face feature parameters comprise shape parameter samples and texture parameter samples.

10. The face focusing device of claim 7, further comprising:

11. The face focusing device of claim 10, wherein the obtaining module is further configured to:

12. The device as claimed in any one of claims 7-11, wherein the focusing module is further configured to:

and focusing the human face based on the target focusing image distance.

13. An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions when executed by the processor implementing the steps of the face focusing method of any one of claims 1-6.