WO2021169668A1

WO2021169668A1 - Image processing method and related device

Info

Publication number: WO2021169668A1
Application number: PCT/CN2021/072461
Authority: WO
Inventors: 颜波
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-02-26
Filing date: 2021-01-18
Publication date: 2021-09-02
Also published as: CN111368678A; CN111368678B

Abstract

Disclosed in the embodiments of the present application are an image processing method and a related device, which are applied to an electronic device. Said method comprises: acquiring a reference face area of a face image, and determining a bounding box of the reference face area; adjusting the bounding box of the reference face area to obtain a bounding box of a standard face area, and determining the standard face area according to the bounding box of the standard face area, the adjustment for the bounding box being used to adjust a rectangular bounding box to a square bounding box; inputting the standard face area into a first neural network model to obtain face key point coordinates of the face image; and inputting sample data comprising the face key point coordinates to a second neural network model to train the second neural network model, the trained second neural network model being a high-precision face key point detection model. The embodiments of the present application help to improve the precision of face key point detection.

Description

Image processing method and related device

Technical field

This application relates to the technical field of mobile terminals, and in particular to an image processing method and related devices.

Background technique

The application field of face key point detection is very wide. Through the face key point detection, the facial features of the face can be accurately located, so that the facial features of the face can be located or adjusted. For example, the face recognition function of mobile devices such as smartphones and tablet computers, and the beauty and makeup functions of the camera when taking pictures require accurate face key detection. Face key point detection based on deep learning. First, the face area is identified from the face image through the face detection network, and then the face area is input to the face key point detection network to obtain the face key points. The diversity of face areas detected by different face images is inconsistent in size and shape, resulting in the inability to obtain more accurate face key points.

Summary of the invention

The embodiments of the present application provide an image processing method and related devices, which are beneficial to improve the accuracy of detecting key points of a human face.

In a first aspect, an embodiment of the present application provides an image processing method, which is characterized in that it is applied to an electronic device, and the method includes:

Acquiring a reference face region of a face image, and determining a bounding box of the reference face region;

The bounding box of the reference face area is adjusted to obtain the bounding box of the standard face area, and the standard face area is determined according to the bounding box of the standard face area, and the adjustment of the bounding box is used to use the rectangle Adjust the bounding box to a square bounding box;

Inputting the standard face region into the first neural network model to obtain face key point coordinates of the face image;

The sample data including the coordinates of the key points of the face is input to a second neural network model to train the second neural network model, wherein the trained second neural network model is a high-precision face key point Check the model.

In a second aspect, an embodiment of the present application provides an image processing device, which is applied to an electronic device, the electronic device includes an eye tracking component; the image processing device includes a processing unit and a communication unit, wherein:

The processing unit is configured to obtain a first face image set through the eye tracking component during the eye tracking calibration process; and to identify the human eyes of the M face images included in the first face image set Area, wherein the M face images are face images captured continuously within a preset time period, and M is a positive integer; Multi-frame fusion of M face images in a face image set is performed to obtain a second face image set including superpixel face images, wherein the second face image set includes N face images, and N Less than M; and used to obtain an eye tracking calculation equation according to the second face image set, and the calculation equation is used to calculate the gaze point of the user during the eye tracking process.

In a third aspect, an embodiment of the present application provides an electronic device, including a controller, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be controlled by the above The above program includes instructions for executing the steps in any method in the first aspect of the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute On the one hand, part or all of the steps described in any method.

In a fifth aspect, the embodiments of the present application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute as implemented in this application. For example, part or all of the steps described in any method of the first aspect. The computer program product may be a software installation package.

It can be seen that, in this embodiment of the application, the electronic device first obtains the reference face area of the face image, and determines the bounding box of the reference face area, and secondly, adjusts the bounding box of the reference face area. Obtain the bounding box of the standard face area, and determine the standard face area according to the bounding box of the standard face area. The adjustment of the bounding box is used to adjust the rectangular bounding box to a square bounding box, and then, the The standard face area is input to the first neural network model to obtain the face key point coordinates of the face image, and finally, the sample data containing the face key point coordinates is input to the second neural network model to compare The second neural network model is trained, wherein the trained second neural network model is a high-precision face key point detection model. Because in the process of adjusting the bounding box of the reference face area to the bounding box of the standard face area, the rectangular bounding box is adjusted to a square bounding box, and the deformation and/or distortion of the face image is avoided, which is beneficial Improve the accuracy of face key point detection.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1A is a schematic flowchart of an image processing method provided by an embodiment of the present application;

FIG. 1B is a schematic diagram of a reference face area provided by an embodiment of the present application;

FIG. 1C is a schematic diagram of a standard face area provided by an embodiment of the present application;

FIG. 1D is a schematic diagram of a bounding box to be adjusted according to an embodiment of the present application;

FIG. 1E is a schematic diagram of another bounding box to be adjusted according to an embodiment of the present application;

FIG. 2 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Fig. 4 is a block diagram of functional units of an image processing device provided by an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the application, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

Electronic devices may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices (such as smart watches, smart bracelets, pedometers, etc.), computing devices, or other processing accuracy connected to a wireless modem. User equipment (User Equipment, UE), mobile station (Mobile Station, MS), terminal equipment (terminal device), and so on. For ease of description, the devices mentioned above are collectively referred to as electronic devices.

The following describes the embodiments of the application in detail.

Traditional face key point detection algorithms are mainly divided into coordinate regression methods and deep learning methods based on neural networks. Since deep learning methods based on neural networks, especially methods based on convolutional neural networks, have achieved very good results in many computer vision tasks, they have also significantly helped improve the performance of face key point detection. Therefore, the current face The key point detection algorithm is mainly based on the deep learning method based on convolutional neural network. The general step of face key point detection based on deep learning is to first detect the face area from a given image through the face detection network, and then input the detected face area into the face key point detection network for face Key point detection. Due to the diversity of faces, the size and shape of the face regions detected from different images are inconsistent, so they need to be adjusted to a uniform size, and then input into the face key point detection network for face key point detection.

In this application, the face area directly detected by the face detection model is called the reference face area, and the face area that is ready to be input into the face key point detection model after adjusting the size is called the standard face area. The shape of the face image is a square. In the prior art, converting the reference face area into a standard face area generally uses a method of forcibly changing the size of the picture to a preset size. The forced transformation has little effect on the image where the reference face area is close to a square. , But for the reference face region that is not a square image, the forced change will cause face deformation and distortion, which will greatly affect the accuracy of the subsequent face key point detection. Therefore, this application proposes a unified method for converting a standard face image into a reference face image, which ensures that the transformation process of all face images is consistent, and reduces the occurrence of face deformation and distortion. , It is helpful to improve the accuracy of face key point detection.

This application proposes an image processing method, where the image processing method is used to improve the accuracy of detecting key points of a human face. The standard face area obtained in this application satisfies the following conditions: the standard face area is a square; the standard face area includes most of the key points of the face; the size of the standard face area is appropriate, and the bounding box of the standard face area The distance to the key points of the face is appropriate; there is a correlation between the bounding box of the standard face area and the bounding box of the reference face area. The face key point detection method provided in this application for realizing the above four conditions can ensure that the face key points obtained by all face images are more accurate, and the transformation process in this application does not use the face key points Related prior information, and the transformed face area is a square, there will be no face deformation and distortion caused by forced transformation, and it has a significant effect on improving the accuracy of face key detection.

Please refer to FIG. 1A. FIG. 1A is a schematic flowchart of an image processing method provided in an embodiment of the present application, which is applied to an electronic device. As shown in the figure, the image processing method includes:

S101: The electronic device obtains a reference face area of a face image, and determines a bounding box of the reference face area.

Among them, the trained face detection model can be used to perform face detection on the face image, so as to determine the reference face area and the bounding box of the reference face area. As shown in Figure 1B, 100 is a face image that includes face key points annotation, 101 is a bounding box of the reference face image, inside the bounding box is the reference face image, the position of face key point annotation 102 corresponds to the face key point. A large number of experiments have proved that although the trained face detection model can detect most faces, including side faces and occluded parts, the reference face area detected by the face detection model has the following problems: Part of the face The key point is not located in the rectangular area corresponding to the bounding box of the reference face area, and the bounding box of the reference face area has a rectangular shape instead of a square, and the position is biased toward the forehead.

S102. The electronic device adjusts the bounding box of the reference face area to obtain the bounding box of the standard face area, and determines the standard face area according to the bounding box of the standard face area, and is directed to the bounding box The adjustment of is used to adjust the rectangular bounding box to a square bounding box.

Among them, the shape of the bounding box of the reference face area is generally rectangular, so the bounding box needs to be added. Adjust from holding to square to get the bounding box of the standard face area. By observing the bounding box of a large number of reference face images, it is found that the height of the bounding box of the reference face image is generally greater than the width. Therefore, when the bounding box of the reference face area is adjusted, the width of the bounding box can be adjusted. After expansion, the expanded margin box can include more key points of the face. In addition, it is found that the general position of the bounding box of the reference face area is higher than that of the face. After adjustment, as shown in Figure 1C, 101 is With reference to the bounding box of the face area, 103 is the bounding box of the standard face area. It can be seen that compared to the bounding box 101, the bounding box 103 includes more face detection key points.

S103: The electronic device inputs the standard face area into the first neural network model to obtain face key point coordinates of the face image.

Among them, through the image processing method provided in this application, the face image corresponding to the standard face area is a square face image, and the square face image does not appear to be deformed and distorted, and more people are included at the same time. Face key points, input the square face image to the first neural network model to obtain the key point coordinates of the face, and use the obtained key point coordinates of the face as sample data, which can be used to train the second neural network model. , The first neural network model is used to locate the key points of the face in the face image, and the second neural network model is used to detect the key points of the face with high precision.

S104. The electronic device inputs the sample data including the coordinates of the key points of the human face into a second neural network model to train the second neural network model, wherein the trained second neural network model is High-precision face key point detection model.

Among them, the second neural network model is a convolutional neural network model. The second neural network model is iteratively trained and optimized through sample data to obtain a high-precision face key point detection model, and the sample data includes the face image The coordinates of the key points of the face.

In a possible example, the adjusting the bounding box of the reference face area to obtain the bounding box of the standard face area includes: determining the height and width of the bounding box of the reference face area; When the width is less than the height, calculate the absolute value of the difference between the width and the height; adjust the width of the bounding box of the reference face area to be consistent with the height, and set the bounding box Move down to obtain the bounding box of the standard face area, wherein the distance of the downward movement is a quarter of the absolute value of the difference.

Among them, determine the height and width of the bounding box of the reference face area. In general, the shape of the bounding box of the reference face area is rectangular and the width is smaller than the height. Therefore, when the width is smaller than the height, the width and height are calculated. The absolute value of the difference. At this time, you can adjust the border of the reference face area and then adjust the width of the frame to be the same as the height, and move the bounding box down to get the bounding box of the standard face area. The distance of the downward movement is A quarter of the absolute value of the difference. By comparing and adjusting the position of the bounding box of the reference face area of a large number of face images and the face image in the reference face area, it is found that when the downward movement distance is one-fourth of the absolute value of the difference, it can be changed. Many face detection key points are contained in the bounding box.

It can be seen that in this example, since the width of the bounding box of the reference face area is less than the height, after adjusting the width to be consistent with the height, a square bounding box can be obtained, and the square bounding box can make the face more critical Points are included. Since the position of the bounding box of the reference face area is usually higher relative to the face, moving the square bounding box down can include more face key points in the lower part of the face, which is beneficial to improve the face key The accuracy of point detection.

In a possible example, the moving the bounding box down to obtain the bounding box of the standard face area includes: determining whether the bounding box moved down is located in the display area of the face image; If not, adjusting the bounding box moved down to obtain the bounding box of the standard face area, and the adjustment is used to make the bounding box located in the display area of the face image.

Wherein, if the bounding box is moved out of the display area of the face image after the bounding box is moved down, the bounding box after moving down needs to be adjusted again. In fact, the bounding box is completely located at all. In the display area of the face image. As shown in FIG. 1D, the bounding box 103 obtained after moving down exceeds the display area of the face image. Therefore, the bounding box in FIG. 1D needs to be adjusted again so that the bounding box 103 is completely located in the display area of the face image. Inside.

It can be seen that in this example, when the face image is at the edge of the display area, moving the bounding box down may cause the bounding box to exceed the face display area. Therefore, after moving the bounding box down, it is necessary to determine the next step. Whether the subsequent bounding box is completely located in the display area of the face image, so when it is detected that the bounding box moved down is not completely located in the display area, the bounding box needs to be adjusted again.

In a possible example, the adjusting the bounding box after moving down to obtain the bounding box of the standard face area includes: detecting that the area of the bounding box after moving down is smaller than the face of the face When calculating the image area, calculate the offset distance and offset direction of the bounding box relative to the face image after moving down; move the bounding box in the opposite direction of the offset direction, so that the moving distance is equal to the The shifted and moved bounding box is located in the display area of the face image.

Among them, when adjusting the bounding box after moving down, the strategy of moving the bounding box is first adopted. By moving the bounding box, the bounding box can be completely translated into the display area of the face image, and the area of the bounding box after moving down is first detected. Whether it is smaller than the face image area, if yes, calculate the offset distance and offset direction of the downward bounding box relative to the face image. As shown in Figure 1E, the offset distance is d and the offset direction is downward. In this case, the bounding box can be moved in the opposite direction of the offset direction, and the moving distance is equal to the offset distance, that is, the bounding box 103 is moved upward, and the moving distance is d. At this time, the bounding box 103 can be located in the display area 100.

It can be seen that in this example, when the bounding box after moving down is not completely within the display area of the face image, first check whether the area of the bounding box is smaller than the area of the face image. If so, you can make the boundary by moving the bounding box. The frame is completely located in the display area, determine the offset direction and offset distance of the bounding box relative to the face image display area, and move the offset distance in the opposite direction of the offset direction to get the face image display area Bounding box inside.

In a possible example, the adjusting the bounding box moved down to obtain the bounding box of the standard face area includes: determining the bounding box moved down and the face image display area The intersection area, wherein the intersection area is a rectangular area; determine the longer side and the shorter side of the intersection area, and use the shorter side as a reference to crop a square area from the intersection area to determine the The bounding box of the square area is the bounding box of the standard face area.

Among them, if the bounding box moved down is not completely located in the display area of the face image, the intersection area between the bounding box moved down and the face image display area can be determined at this time. It can be seen that the intersection area must be a rectangular area. The longer side and shorter side of the intersecting area, and using the shorter side as the benchmark, crop a square area from the intersecting area. The side length of the square area is equal to the shorter side length. At this time, a square in the intersecting area is obtained. Area, the square area can be used as a standard face area.

It can be seen that in this example, if the bounding box after moving down is not completely within the display area of the face image, in order to obtain a square bounding box of the standard face area, you can choose to crop a square area in the intersection area at this time. And use the square area as the standard face area. Through this cropping method, more key points of face detection can be included in the bounding box of the obtained standard face image area.

In a possible example, the cutting a square area from the intersecting area based on the shorter side includes: detecting whether the intersecting area exists and the boundary of the face image display area, and moving down If the bounding box overlaps at the same time; if yes, cut the square area with the overlapped side as the side length, so that the side length of the cut square area is equal to the length of the shorter side of the intersecting area; if not, start from the In the intersection area, a square area is symmetrically cut out, so that the center of the cut out square area coincides with the center of the intersection area.

Among them, when it is detected that the intersecting area between the lower bounding box and the face area display area, there is the boundary of the face image display area and the edges that coincide with the lower bounding box at the same time, then the coincident The side is taken as the side length, and a square area is cut out. The side length of the cut square area is equal to the length of the shorter side of the intersecting area. If there are no overlapping sides, a square area is symmetrically cut out from the intersecting area and cut out The center of the square area coincides with the center of the intersecting area.

It can be seen that in this example, a square area is cut out from the intersection area as the standard face area. First, it is detected whether there is an edge in the intersection area that coincides with the boundary of the face image display area and the bounding box after moving down. , You can crop a square area based on the coincident edge, so as to ensure that the bounding box of the cropped square area overlaps with the following bounding boxes as much as possible, which helps to ensure that the square area includes more face detection keys. point.

In a possible example, the method further includes: when it is detected that the face image includes multiple faces, dividing the face image into multiple face image regions according to the number of faces; The priority of the personal face is to determine the reference face area of the multiple face image areas in sequence.

Among them, when it is detected that the face image includes multiple faces, the face image can be divided into multiple face image regions according to the number of faces, and face key points are performed on each of the multiple face regions in turn. For detection, for example, the reference face regions of the multiple face image regions can be sequentially determined according to the priority of the multiple faces, and then the face key points of the multiple face image regions can be determined.

It can be seen that in this example, when the face image is a group photo of multiple people, the face image will include multiple faces. At this time, the face image can be divided into multiple face image areas, and each face image area Face key point detection is performed on the face image in, so that the key point of each face can be obtained.

Consistent with the embodiment shown in FIG. 1A, please refer to FIG. 2. FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application, which is applied to an electronic device. As shown in the figure, the image processing method includes:

S201: The electronic device obtains a reference face area of a face image, and determines a bounding box of the reference face area.

S202: The electronic device determines the height and width of the bounding box of the reference face area.

S203: When detecting that the width is less than the height, the electronic device calculates the absolute value of the difference between the width and the height.

S204: The electronic device adjusts the width of the bounding box of the reference face area to be consistent with the height, and moves the bounding box downward to obtain the bounding box of the standard face area, wherein The distance moved downward is a quarter of the absolute value of the difference.

S205: The electronic device determines a standard face area according to the bounding box of the standard face area.

S206: The electronic device inputs the standard face area into the first neural network model to obtain face key point coordinates of the face image.

S207. The electronic device inputs the sample data including the coordinates of the key points of the human face into a second neural network model to train the second neural network model, wherein the trained second neural network model is High-precision face key point detection model.

In addition, since the width of the bounding box of the reference face area is smaller than the height, after adjusting the width to be consistent with the height, a square bounding box can be obtained, and the square bounding box can include more key points of the face. Since the position of the bounding box of the reference face area is usually higher relative to the face, moving the square bounding box down can include more face key points under the face, which is beneficial to improve the accuracy of face key point detection Spend.

Consistent with the embodiment shown in FIG. 1A and FIG. 2, please refer to FIG. 3. FIG. 3 is a schematic structural diagram of an electronic device 300 provided by an embodiment of the present application. The electronic device 300 runs one or more applications Programs and operating systems, as shown in the figure, the electronic device 300 includes a processor 310, a memory 320, a communication interface 330, and one or more programs 321, wherein the one or more programs 321 are stored in the memory 320 And is configured to be executed by the processor 310, and the one or more programs 321 include instructions for executing the following steps;

In a possible example, in terms of adjusting the bounding box of the reference face area to obtain the bounding box of the standard face area, the instructions in the program are specifically used to perform the following operations: determining the reference face area The height and width of the bounding box of the face area; when it is detected that the width is smaller than the height, the absolute value of the difference between the width and the height is calculated; the width of the bounding box of the reference face area is adjusted In order to be consistent with the height, the bounding box is moved downward to obtain the bounding box of the standard face area, wherein the distance of the downward movement is a quarter of the absolute value of the difference.

In a possible example, in terms of moving the bounding box down to obtain the bounding box of the standard face area, the instructions in the program are specifically used to perform the following operations: judging the moving down Whether the bounding box is located in the display area of the face image; if not, adjust the bounding box after moving down to obtain the bounding box of the standard face area, and the adjustment is used to make the bounding box be located at the In the display area of the face image.

In a possible example, in terms of adjusting the bounding box after moving down to obtain the bounding box of the standard face area, the instructions in the program are specifically used to perform the following operations: When the area of the moved bounding box is smaller than the area of the face image, calculate the offset distance and direction of the moved down bounding box relative to the face image; according to the opposite of the offset direction Move the bounding box in a direction such that the moving distance is equal to the offset distance and the moved bounding box is located in the display area of the face image.

In a possible example, in terms of adjusting the bounding box after moving down to obtain the bounding box of the standard face region, the instructions in the program are specifically used to perform the following operations: The intersection area between the bounding box and the face image display area, wherein the intersection area is a rectangular area; the longer side and the shorter side of the intersection area are determined, and the shorter side is used as a reference A square area is cropped from the intersection area, and the bounding box of the square area is determined to be the bounding box of the standard face area.

In a possible example, in terms of cropping a square area from the intersecting area based on the shorter side, the instructions in the program are specifically used to perform the following operations: detecting whether the intersecting area exists The edge that coincides with the boundary of the face image display area and the bounding box moved down at the same time; if it is, the square area is cropped with the overlapping edge as the side length, so that the length of the side of the cropped square area is equal to the intersection The length of the shorter side of the area; if not, a square area is symmetrically cut out from the intersection area, so that the center of the cut out square area coincides with the center of the intersection area.

In a possible example, the instructions in the program are specifically used to perform the following operations: when it is detected that the face image includes multiple faces, the face image is divided into multiple faces according to the number of faces Image area; according to the priority of the multiple faces, sequentially determine the reference face area of the multiple face image areas.

In a possible example, the first neural network model is used to locate the key points of the face in the face image, and the second neural network model is used to detect the key points of the face with high precision.

In a possible example, the number of face detection key points included in the square bounding box obtained after adjustment is greater than the number of face detection key points included in the rectangular bounding box.

The foregoing mainly introduces the solution of the embodiment of the present application from the perspective of the execution process on the method side. It can be understood that, in order to implement the above-mentioned functions, an electronic device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The embodiment of the present application may divide the electronic device into functional units according to the foregoing method examples. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one control unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.

FIG. 4 is a block diagram of the functional unit composition of a device 400 involved in an embodiment of the present application. The image processing device 400 is applied to electronic equipment. The image processing device 400 includes a processing unit 401 and a communication unit 402, wherein:

The processing unit 401 is configured to obtain a reference face area of a face image through the communication unit 402, and determine a bounding box of the reference face area; and to perform processing on the bounding box of the reference face area Adjust to obtain a bounding box of the standard face area, and determine the standard face area according to the bounding box of the standard face area, the adjustment to the bounding box is used to adjust the rectangular bounding box to a square bounding box; and Inputting the standard face area into the first neural network model to obtain the face key point coordinates of the face image; and for inputting sample data including the face key point coordinates to the second neural network The network model is used to train the second neural network model, wherein the trained second neural network model is a high-precision face key point detection model.

In a possible example, in terms of adjusting the bounding box of the reference face area to obtain the bounding box of the standard face area, the processing unit 401 is specifically configured to: determine the bounding box of the reference face area The height and width of the bounding box; and when it is detected that the width is smaller than the height, calculating the absolute value of the difference between the width and the height; The width is adjusted to be consistent with the height, and the bounding box is moved downward to obtain the bounding box of the standard face area, wherein the distance of the downward movement is a quarter of the absolute value of the difference .

In a possible example, in the aspect of moving the bounding box down to obtain the bounding box of the standard face area, the processing unit 401 is specifically configured to: determine whether the bounding box after moving down is located in The display area of the face image; if not, adjust the bounding box after moving down to obtain the bounding box of the standard face area, and the adjustment is used to make the bounding box located in the face image Within the display area.

In a possible example, in terms of adjusting the bounding box after moving down to obtain the bounding box of the standard face area, the processing unit 401 is specifically configured to: When the area of the bounding box is smaller than the area of the face image, calculating the offset distance and the offset direction of the bounding box moved down relative to the face image; and used to calculate the offset direction according to the opposite direction of the offset direction Move the bounding box so that the moving distance is equal to the offset distance and the moved bounding box is located in the display area of the face image.

In a possible example, in terms of adjusting the bounding box after moving down to obtain the bounding box of the standard face region, the processing unit 401 is specifically configured to: determine the bounding box after moving down The intersection area between the frame and the face image display area, wherein the intersection area is a rectangular area; A square area is cropped in the intersection area, and the bounding box of the square area is determined to be the bounding box of the standard face area.

In a possible example, in the aspect of cropping a square area from the intersecting area based on the shorter side, the processing unit 401 is specifically configured to: detect whether the intersecting area exists and the person The boundary of the face image display area and the edge that coincides with the lower bounding box at the same time; if so, the square area is cropped with the overlapped side as the side length, so that the side length of the cropped square area is equal to the shorter side of the intersecting area If not, cut out a square area symmetrically from the intersection area, so that the center of the cut out square area coincides with the center of the intersection area.

In a possible example, the processing unit 401 is specifically configured to: when detecting that the face image includes multiple faces, divide the face image into multiple face image regions according to the number of faces; The priority of the multiple human faces is determined in turn from the reference face regions of the multiple human face image regions.

The electronic device may further include a storage unit 403, the processing unit 401 and the communication unit 402 may be a controller or a processor, and the storage unit 403 may be a memory.

An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method as recorded in the above method embodiment , The above-mentioned computer includes a mobile terminal.

The embodiments of the present application also provide a computer program product. The above-mentioned computer program product includes a non-transitory computer-readable storage medium storing a computer program. Part or all of the steps of the method. The computer program product may be a software installation package, and the above-mentioned computer includes a mobile terminal.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps can be performed in other order or at the same time. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one control unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory. A number of instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the foregoing methods of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable memory, and the memory can include: a flash disk , Read-only memory (English: Read-Only Memory, abbreviation: ROM), random access device (English: Random Access Memory, abbreviation: RAM), magnetic disk or optical disc, etc.

The embodiments of the application are described in detail above, and specific examples are used in this article to illustrate the principles and implementation of the application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the application; at the same time, for Those of ordinary skill in the art, based on the idea of the application, will have changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation to the application.

Claims

An image processing method, characterized in that it is applied to an electronic device, and the method includes:

Acquiring a reference face region of a face image, and determining a bounding box of the reference face region;

The bounding box of the reference face area is adjusted to obtain the bounding box of the standard face area, and the standard face area is determined according to the bounding box of the standard face area, and the adjustment of the bounding box is used to use the rectangle Adjust the bounding box to a square bounding box;

Inputting the standard face region into the first neural network model to obtain face key point coordinates of the face image;

The sample data including the coordinates of the key points of the face is input to a second neural network model to train the second neural network model, wherein the trained second neural network model is a high-precision face key point Check the model.
The method according to claim 1, wherein the adjusting the bounding box of the reference face area to obtain the bounding box of the standard face area comprises:

Determining the height and width of the bounding box of the reference face region;

When it is detected that the width is smaller than the height, calculating the absolute value of the difference between the width and the height;

The width of the bounding box of the reference face area is adjusted to be consistent with the height, and the bounding box is moved downward to obtain the bounding box of the standard face area, wherein the distance of the downward movement is A quarter of the absolute value of the difference.
The method according to claim 2, wherein the moving the bounding box downward to obtain the bounding box of the standard face area comprises:

Judging whether the bounding box moved down is located in the display area of the face image;

If not, adjusting the bounding box moved down to obtain the bounding box of the standard face area, and the adjustment is used to make the bounding box located in the display area of the face image.
The method according to claim 3, wherein the adjusting the bounding box moved down to obtain the bounding box of the standard face region comprises:

When it is detected that the area of the bounding box after moving down is smaller than the area of the face image, calculating the offset distance and the offset direction of the bounding box after moving down relative to the face image;

Move the bounding box in a direction opposite to the offset direction, so that the moving distance is equal to the offset distance and the moved bounding box is located in the display area of the face image.
The method according to claim 3, wherein the adjusting the bounding box moved down to obtain the bounding box of the standard face region comprises:

Determining the intersection area between the downwardly shifted bounding box and the face image display area, wherein the intersection area is a rectangular area;

Determine the longer side and the shorter side of the intersecting area, and crop a square area from the intersecting area based on the shorter side, and determine that the bounding box of the square area is that of the standard face area Bounding box.
The method according to claim 5, wherein the cropping a square area from the intersecting area based on the shorter side comprises:

Detecting whether the intersection area has an edge that coincides with the boundary of the face image display area and the bounding box after being moved down at the same time;

If yes, cut the square area with the coincident side as the side length, so that the side length of the cut square area is equal to the length of the shorter side of the intersecting area;

If not, a square area is symmetrically cut out from the intersection area, so that the center of the cut out square area coincides with the center of the intersection area.
The method according to any one of claims 1-6, wherein the method further comprises:

When it is detected that the face image includes multiple faces, dividing the face image into multiple face image regions according to the number of faces;

According to the priority of the multiple faces, the reference face regions of the multiple face image regions are sequentially determined.
The method according to any one of claims 1, wherein the first neural network model is used for locating key points of the face in the face image, and the second neural network model is used for high-precision detection of the face key point.
The method according to any one of claims 1, wherein the number of face detection key points included in the square bounding box obtained after adjustment is greater than the number of face detection key points included in the rectangular bounding box.
An image processing device, characterized in that it is applied to electronic equipment, the image processing device includes a processing unit and a communication unit, wherein:

The processing unit is configured to obtain a reference face area of a face image through the communication unit, and determine a bounding box of the reference face area; and to adjust the bounding box of the reference face area, Obtain the bounding box of the standard face area, and determine the standard face area according to the bounding box of the standard face area, the adjustment of the bounding box is used to adjust the rectangular bounding box to a square bounding box; and Inputting the standard face area to the first neural network model to obtain the face key point coordinates of the face image; and inputting sample data including the face key point coordinates to the second neural network model The second neural network model is trained, wherein the trained second neural network model is a high-precision face key point detection model.
The image processing device according to claim 10, wherein in the aspect of adjusting the bounding box of the reference face area to obtain the bounding box of the standard face area, the processing unit is specifically configured to: determine The height and width of the bounding box of the reference face area; when it is detected that the width is smaller than the height, the absolute value of the difference between the width and the height is calculated; and the bounding box of the reference face area The width of is adjusted to be consistent with the height, and the bounding box is moved downward to obtain the bounding box of the standard face region, wherein the distance of the downward movement is a quarter of the absolute value of the difference one.
The image processing device according to claim 11, wherein, in terms of moving the bounding box down to obtain the bounding box of the standard face area, the processing unit is specifically configured to: determine that the bounding box is moved down Whether the bounding box is located in the display area of the face image; if not, adjusting the bounding box after moving down to obtain the bounding box of the standard face area, and the adjustment is used to make the boundary The frame is located in the display area of the face image.
The image processing device according to claim 12, wherein, in terms of adjusting the bounding box moved down to obtain the bounding box of the standard face region, the processing unit is specifically configured to: When it is detected that the area of the bounding box after moving down is smaller than the area of the face image, calculate the offset distance and direction of the bounding box after moving down relative to the face image; according to the offset The bounding box is moved in the opposite direction of the direction, so that the moving distance is equal to the offset distance and the moved bounding box is located in the display area of the face image.
The image processing device according to claim 12, wherein, in terms of adjusting the bounding box moved down to obtain the bounding box of the standard face region, the processing unit is specifically configured to: determine The intersection area between the lowered bounding box and the face image display area, wherein the intersection area is a rectangular area; the longer side and the shorter side of the intersection area are determined, and the shorter side A square area is cropped from the intersection area based on the edge, and the bounding box of the square area is determined to be the bounding box of the standard face area.
The image processing device according to claim 14, wherein in the aspect of cropping a square area from the intersection area based on the shorter side, the processing unit is specifically configured to: detect the intersection Whether the area has an edge that coincides with the boundary of the face image display area and the bounding box after moving down at the same time; if so, the square area is cropped with the overlapped edge as the side length, so that the side length of the cropped square area is equal to The length of the shorter side of the intersection area; if not, a square area is symmetrically cut out from the intersection area, so that the center of the cut out square area coincides with the center of the intersection area.
The image processing device according to any one of claims 10-15, wherein the processing unit is specifically configured to: when it is detected that the face image includes multiple faces, calculate the number of faces according to the number of faces. The face image is divided into multiple face image regions; and the reference face regions of the multiple face image regions are sequentially determined according to the priority of the multiple face images.
The image processing device according to claim 10, wherein the first neural network model is used to locate the key points of the face in the face image, and the second neural network model is used to detect the key points of the face with high precision. point.
The image processing device according to claim 10, wherein the number of face detection key points included in the square bounding box obtained after adjustment is greater than the number of face detection key points included in the rectangular bounding box.
An electronic device, characterized by comprising a processor, a memory, a communication interface, and one or more programs, the one or more programs are stored in the memory and configured to be executed by the processor, The program includes instructions for executing the steps in the image processing method according to any one of claims 1-9.
A computer-readable storage medium, characterized by storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the image processing method according to any one of claims 1-9.