WO2019033575A1

WO2019033575A1 - Electronic device, face tracking method and system, and storage medium

Info

Publication number: WO2019033575A1
Application number: PCT/CN2017/108760
Authority: WO
Inventors: 戴磊
Original assignee: 平安科技（深圳）有限公司
Priority date: 2017-08-17
Filing date: 2017-10-31
Publication date: 2019-02-21
Also published as: CN107633208B; CN107633208A

Abstract

An electronic device, a face tracking method and system, and a storage medium. The method comprises: a face position determining step: obtaining a time sequence of photographed face images, selecting two adjacent frames of face images from the time sequence, and determining face regions in the two adjacent frames of face images (S1); a similarity calculating step: calculating the similarity of faces in the two adjacent frames of face images based on X and Y coordinate values of center points of the face regions in the two adjacent frames of face images, and the heights H and widths W of the face regions (S2); and a face tracking determining step: performing face tracking based on the similarity of the faces in the two adjacent frames of face images (S3). The method can accurately track a face in case of transient shielding of the face or in absence of face detection of a few people.

Description

Electronic device, method and system for face tracking, and storage medium

Priority claim

This application is based on the priority of the Paris Convention for the Chinese patent application entitled "Electronic Device, Face Tracking Method and Storage Medium", filed on August 17, 2017, with the application number CN 201710709124.0, the overall content of the Chinese patent application It is incorporated herein by reference.

Technical field

The present application relates to the field of image processing technologies, and in particular, to an electronic device, a method and system for tracking a face, and a storage medium.

Background technique

At present, when tracking the face of the same person, the general method is based on the position of the face, and the face of the same person is judged to be the face of the same person when the face of the current frame is closest to the coordinates of the face center of the previous frame. . Since only the x and y coordinates are used, and the depth information is not used, when a short occlusion occurs during the tracking process, or when a certain number of faces are not detected, it may cause the near face to be mistaken for The follow-up (or vice versa) of a distant face causes an error in face tracking.

Summary of the invention

The purpose of the present application is to provide an electronic device, a method and system for tracking a face, and a storage medium, which are intended to accurately perform face tracking when a short occlusion occurs or a small number of face detections are missing.

To achieve the above object, the present application provides an electronic device including a memory and a processor connected to the memory, wherein the memory stores a face tracking system operable on the processor, When the face tracking system is executed by the processor, the following steps are implemented:

a face position determining step: acquiring a face image of a time series of the shooting, taking two adjacent frames of the face image from the time sequence, and determining a face region in the adjacent two frames of the face image;

The similarity calculation step: calculating the adjacent two frames of face images according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W value of the face region The similarity of the face in the face;

Face tracking judging step: performing face tracking based on the similarity of faces in adjacent two frames of face images.

To achieve the above object, the present application further provides a method for face tracking, and the method for face tracking includes:

The similarity calculation step: calculating the adjacent two frames according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W of the face region The similarity of faces in the image;

To achieve the above objective, the present application provides a face tracking system, and the face tracking system includes:

a determining module, configured to obtain a time-series face image of the shooting, take two adjacent frames of the face image from the time series, and determine a face region in the adjacent two frames of the face image;

a calculation module, configured to calculate the adjacent two frames of the face image according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W value of the face region The similarity of the face in the face;

The tracking module is configured to perform face tracking based on the similarity of faces in the adjacent two frames of the face image.

The present application also provides a computer readable storage medium on which a face tracking system is stored, and when the face tracking system is executed by a processor, the steps are implemented:

The beneficial effects of the present application are: in the face tracking, in the face tracking, in addition to the x and y coordinates, the face size is further increased as the judgment basis of the same target face, because the person is in the three directions of x, y, and z. Generally, there is no high moving speed, so the face size of the same person does not change greatly. Therefore, for each face of the T-1 frame, the most similarity is found in all faces of the T frame. As a follow-up to his face, the face reduces the possibility of erroneous tracking in the event of a short occlusion or a small number of face detections, enabling accurate face tracking.

DRAWINGS

1 is a schematic diagram of an optional application environment of each embodiment of the present application;

2 is a schematic flow chart of an embodiment of a method for tracking a face of an applicant.

Detailed ways

In order to make the objects, technical solutions, and advantages of the present application more comprehensible, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

It should be noted that the descriptions referring to “first”, “second” and the like in the present application are only used for description. The purpose is not to be construed as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" and "second" may include at least one of the features, either explicitly or implicitly. In addition, the technical solutions between the various embodiments may be combined with each other, but must be based on the realization of those skilled in the art, and when the combination of the technical solutions is contradictory or impossible to implement, it should be considered that the combination of the technical solutions does not exist. Nor is it within the scope of protection required by this application.

Referring to FIG. 1 , it is a schematic diagram of an application environment of a preferred embodiment of the method for face tracking of the applicant. The application environment diagram includes an electronic device 1 and a camera device 2. The electronic device 1 can perform data interaction with the camera device 2 through a suitable technology such as a network or a near field communication technology.

The imaging device 2 may be a camera including a TUBE vacuum tube sensor, a CCD (Charge Coupled Device) charge coupling module sensor, or a CMOS metal oxide semiconductor sensor, and is not limited thereto. The camera device 2 includes one or more devices installed in a specific place (for example, an office place and a monitoring area), and captures a video in real time for a target entering the specific place, and transmits the captured video to the electronic device 1 in real time through the network.

The electronic device 1 is an apparatus capable of automatically performing numerical calculation and/or information processing in accordance with an instruction set or stored in advance. The electronic device 1 may be a computer, a single network server, a server group composed of multiple network servers, or a cloud-based cloud composed of a large number of hosts or network servers, where cloud computing is a type of distributed computing. A super virtual computer consisting of a group of loosely coupled computers.

In this embodiment, the electronic device 1 may include, but is not limited to, a memory 11 communicably connected to each other through a system bus, a processor 12, and a network interface 13, and the memory 11 stores a face tracking executable on the processor 12. system. It should be noted that FIG. 1 only shows the electronic device 1 having the components 11-13, but it should be understood that not all illustrated components may be implemented, and more or fewer components may be implemented instead.

The memory 11 may further include a memory and at least one type of readable storage medium, and the memory provides a cache for the operation of the electronic device 1. The readable storage medium can also be used to store a real-time captured face image and a library of face image samples received by the electronic device 1. The readable storage medium may be, for example, a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electric A non-volatile storage medium that can erase a programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the readable storage medium may also be an external storage device of the electronic device 1. For example, a plug-in hard disk equipped with an electronic device 1, a smart memory card (SMC), a Secure Digital (SD) card, a flash card, and the like. In this embodiment, the memory 11 can also be used to store an operating system installed in the electronic device 1 and various types of application software, such as a program code of a face tracking system in an embodiment of the present application.

The processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. This treatment The device 12 is generally used to control the overall operation of the electronic device 1, for example, performing control and processing related to data interaction or communication with the camera device 2. In this embodiment, the processor 12 is configured to run program code or process data stored in the memory 11, such as running a face tracking system or the like.

The network interface 13 may comprise a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the electronic device 1 and other electronic devices. In this embodiment, the network interface 13 is mainly used to connect the electronic device 1 with one or more camera devices 2 to establish a data transmission channel and a communication connection.

The face tracking system is stored in the memory 11 and includes at least one computer readable instruction stored in the memory 11, the at least one computer readable instruction being executable by the processor 12 to implement the methods of various embodiments of the present application. And the at least one computer readable instruction can be divided into different logic modules according to different functions implemented by the various parts thereof. The embodiment includes a determining module, a calculating module and a tracking module.

In an embodiment, when the face tracking system is executed by the processor 12, the following steps are implemented:

Wherein, in each frame image of the dynamic video captured in real time, the image including the face image is selected based on the feature of the face, and the selected image is used as a face image of a time series.

Specifically, the method of selecting an image including a face image based on a feature of a face includes: based on a traditional method (eg, a contour rule based on a face, an organ distribution rule, a symmetry rule or a motion rule, etc.), based on a geometric feature Methods (such as using machine learning to find facial features), methods based on correlation matching (such as template matching or iso-strength methods, etc.), representation-based methods (such as using statistical analysis and machine learning techniques to find faces) And related features of non-face images), methods based on statistical theory (such as neural networks or support vector machines, etc.).

In this embodiment, for the face image of the time series, two adjacent frames of the face image are acquired, and the face region is determined in the adjacent two frames of the face image. The face area can be large or small. For a face image taken in a distant scene, the face area is small, and for a close-up shot face image, the face area is large. The face area is a minimum area including a human face, and is preferably a rectangular area including a human face. Of course, it may be an area including a human face of other shapes, such as a circular area, and is not limited thereto.

In this embodiment, the similarity calculation formula of the adjacent two frames of the face image is as follows:

The S _i,j is a similarity, and the w _x , w _y , w _w , w _h are the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j, respectively. The weights, w _x , w _y , w _w , w _h ∈[0,1], the weights may be the same or different, where:

The distance between the face i and the center point of the face j in the x direction;

The distance between the face i and the center point of the face j in the y direction;

The difference in width between face i and face j;

The difference in height between face i and face j.

Preferably, the weights w _x , w _y , w _w , w _h of the adjacent two frames of the face i, the x-direction distance, the y-direction distance, the width difference, and the height difference of the face j are both 0.25.

For the face of the same person, the closer the camera is, the larger the proportion of the face area captured in the image, that is, the larger the product of the height H of the face area and the width W value, the farther away from the camera is taken The smaller the proportion of the area occupied by the face area in the image, that is, the smaller the product of the height H of the face area and the width W value, therefore, in this embodiment, the depth information of the image (ie, the change in the z direction) passes. The size of the face region in the real-time image is reflected, and the similarity of the face is calculated based on the x, y, and z directions of the face image.

Compared with the prior art, in the face tracking, in the face tracking, in addition to the x and y coordinates, the face size is further increased as the judgment basis of the same target face, because the person is in x, y, and z. The direction generally does not have a very high moving speed, so the face size of the same person does not change greatly. Therefore, for each face of the T-1 frame, the similarity is the largest among all the faces of the T frame. As a follow-up to his follow-up, this algorithm reduces the possibility of false tracking in the event of a short occlusion or a small number of face detections, enabling accurate face tracking.

In a preferred embodiment, on the basis of the foregoing embodiment of FIG. 1, the face tracking determining step includes: when the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, It is determined that the face in the adjacent two frames of the face image is the same person's face.

In this embodiment, when the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, preferably, the preset threshold is 0.85, and the face in the adjacent two frames of the face image is determined as The face of the same person.

When each of the adjacent two frames of the face image identifies a face region, it is directly determined according to the calculated similarity value whether the face in the adjacent two frames of the face image is the same person. human face. In other embodiments, when there are more than one face region recognized from the image in the adjacent two frames of the face image (for example, two or three face regions are recognized in some images), the formula is calculated according to the similarity degree. Calculating the similarity value of each face region in the two images and each face region in the other face image, and determining the matching faces in the two images according to the similarity value, that is, the similarity of the faces is greater than A face equal to the preset threshold is the face of the same person.

As shown in FIG. 2, FIG. 2 is a schematic flowchart of an embodiment of a method for tracking a face of an applicant, and the method for tracking a face includes the following steps:

S1, a face position determining step: acquiring a face image of a time series of the shooting, taking two adjacent frames of the face image from the time series, and determining a face region in the adjacent two frames of the face image;

S2, the similarity calculation step: calculating the adjacent two frames according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W value of the face region The similarity of faces in the face image;

S3. The face tracking judging step: performing face tracking based on the similarity of the faces in the adjacent two frames of the face image.

The difference in width between face i and face j;

The difference in height between face i and face j.

For the same person's face, the closer to the camera, the captured face area is occupied by the image. The larger the area ratio, that is, the larger the product of the height H of the face region and the width W value, the farther away from the camera, the smaller the proportion of the captured face area in the image, that is, the height H of the face region and The smaller the product of the width W value is, therefore, in this embodiment, the depth information of the image (ie, the change in the z direction) is reflected by the size of the face region in the real-time image, based on the x, y, and z directions of the face image. Calculate the similarity of faces.

In a preferred embodiment, on the basis of the foregoing embodiment of FIG. 2, the face tracking determining step includes: when the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, It is determined that the face in the adjacent two frames of the face image is the same person's face.

The present application also provides a computer readable storage medium storing a face tracking system on a computer readable storage medium, the steps of the method for implementing face tracking described above when the face tracking system is executed by a processor.

The serial numbers of the embodiments of the present application are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is better. Implementation. Based on such understanding, the technical solution of the present application, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, The optical disc includes a number of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present application.

The above is only a preferred embodiment of the present application, and is not intended to limit the scope of the patent application, and the equivalent structure or equivalent process transformations made by the specification and the drawings of the present application, or directly or indirectly applied to other related technical fields. The same is included in the scope of patent protection of this application.

Claims

An electronic device, comprising: a memory and a processor coupled to the memory, wherein the memory stores a face tracking system operable on the processor, the face tracking The system implements the following steps when executed by the processor:

a face position determining step: acquiring a face image of a time series of the shooting, taking two adjacent frames of the face image from the time sequence, and determining a face region in the adjacent two frames of the face image;

The similarity calculation step: calculating the adjacent two frames of face images according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W value of the face region The similarity of the face in the face;

Face tracking judging step: performing face tracking based on the similarity of faces in adjacent two frames of face images.
The electronic device according to claim 1, wherein the similarity calculation step comprises:

The S i,j is a similarity, and the w x , w y , w w , w h are the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j, respectively. Weight, w x , w y , w w , w h ∈[0,1], where:

The distance between the face i and the center point of the face j in the x direction;

The distance between the face i and the center point of the face j in the y direction;

The difference in width between face i and face j;

The difference in height between face i and face j.
The electronic device according to claim 2, wherein the weights of the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j are both 0.25.
The electronic device according to any one of claims 1 to 3, wherein the face tracking determining step comprises:

When the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, the face in the adjacent two frames of the face image is determined to be the same person's face.
The electronic device of claim 4, wherein the predetermined threshold is 0.85.
A method for face tracking, characterized in that the method for face tracking includes:

a face position determining step: acquiring a face image of a time series of the shooting, taking two adjacent frames of the face image from the time sequence, and determining a face region in the adjacent two frames of the face image;

The similarity calculation step: calculating the adjacent two frames according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W of the face region The similarity of faces in the image;

Face tracking judging step: performing face tracking based on the similarity of faces in adjacent two frames of face images.
The method of face tracking according to claim 6, wherein the similarity calculation step comprises:

The S i,j is a similarity, and the w x , w y , w w , w h are the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j, respectively. Weight, w x , w y , w w , w h ∈[0,1], where:

The distance between the face i and the center point of the face j in the x direction;

The distance between the face i and the center point of the face j in the y direction;

The difference in width between face i and face j;

The difference in height between face i and face j.
The method of claim 7 according to claim 7, wherein the weights of the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j are both 0.25.
The method for face tracking according to any one of claims 6 to 8, wherein the face tracking determining step comprises:

When the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, the face in the adjacent two frames of the face image is determined to be the same person's face.
The method of face tracking according to claim 9, wherein the preset threshold is 0.85.
A face tracking system, characterized in that the face tracking system comprises:

a determining module, configured to obtain a time-series face image of the shooting, take two adjacent frames of the face image from the time series, and determine a face region in the adjacent two frames of the face image;

a calculation module, configured to calculate the adjacent two frames of the face image according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W value of the face region The similarity of the face in the face;

The tracking module is configured to perform face tracking based on the similarity of faces in the adjacent two frames of the face image.
The face tracking system according to claim 11, wherein the calculation module is specifically configured to:

The S i,j is a similarity, and the w x , w y , w w , w h are the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j, respectively. Weight, w x , w y , w w , w h ∈[0,1], where:

The distance between the face i and the center point of the face j in the x direction;

The distance between the face i and the center point of the face j in the y direction;

The difference in width between face i and face j;

The difference in height between face i and face j.
The face tracking system according to claim 12, wherein the weights of the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j are both 0.25.
The face tracking system according to any one of claims 11 to 13, wherein the tracking module is specifically configured to:

When the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, the face in the adjacent two frames of the face image is determined to be the same person's face.
The face tracking system of claim 14, wherein the predetermined threshold is 0.85.
A computer readable storage medium, wherein the computer readable storage medium stores a face tracking system, and when the face tracking system is executed by the processor, the steps are:

a face position determining step: acquiring a face image of a time series of the shooting, taking two adjacent frames of the face image from the time sequence, and determining a face region in the adjacent two frames of the face image;

The similarity calculation step: calculating the adjacent two frames of face images according to the X and Y coordinate values of the center point of the face region in the adjacent two frames of the face image, and the height H and the width W value of the face region The similarity of the face in the face;

Face tracking judging step: performing face tracking based on the similarity of faces in adjacent two frames of face images.
The computer readable storage medium of claim 16, wherein the similarity calculation step comprises:

The S i,j is a similarity, and the w x , w y , w w , w h are the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j, respectively. Weight, w x , w y , w w , w h ∈[0,1], where:

The distance between the face i and the center point of the face j in the x direction;

The distance between the face i and the center point of the face j in the y direction;

The difference in width between face i and face j;

The difference in height between face i and face j.
The computer readable storage medium according to claim 17, wherein the weights of the x-direction distance, the y-direction distance, the width difference, and the height difference of the adjacent two frames of the face i and the face j are both 0.25.
The computer readable storage medium according to any one of claims 16 to 18, wherein the face tracking determining step comprises:

When the similarity of the face in the adjacent two frames of the face image is greater than or equal to a preset threshold, the face in the adjacent two frames of the face image is determined to be the same person's face.
The computer readable storage medium of claim 19, wherein the predetermined threshold is 0.85.